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Abstract 


The  work  performed  during  the  first,  year  of  the  contract  has  resulted  in  the  definition 


and  implementation  of  an  experimental  facility  that  emulates  the  behavior  of  a single 


node  in  an  integrated  circuit-packet-switching  communications  network.  There  are 


several  major  components  in  this  study:  We  have  identified  the  functional 


characteristics  of  the  system,  we  have  developed  and  implemented  a task 


decomposition  that  performs  the  different  functions  of  an  integrated  switch,  we  have 


developed  theoretical  models  of  performance  for  the  switch,  and  finally,  we  have 


developed  a reliability  study  of  the  the  hardware  components  of  the  system.  The 


report  does  not  attempt  to  present  results  as  a closed  set  of  corKlusions.  The  main 


product  of  the  first  year  has  been  the  definition  of  the  problem  task,  the  development 


of  the  experimental  facility,  and  performing  related  theoretical  studies;  muc>i  work 


remains  to  be  done  in  both  the  experimental  and  the  theoretical  aspects  of  the  project. 


These  will  be  the  object  of  the  second  year  contract.  The  continuation  effort  will  not 


only  elaborate  on  the  first  year  results  but  will  also  expand  into  other  areas.  These 


areas  will  be  identified  throughout  the  report. 


Chapter  1 describes  the  characteristics  of  the  integrated  switch  and  the  experimental 


facility  implemented  on  C.mmp.  This  chapter  presents  the  results  of  running  several 


experiments  with  various  mixes  of  Class  I and  Class  II  traffic.  Chapter  II  describes  the 


analytical  models  for  the  integrated  switch.  Chapter  III  describes  the  reliability 


models. 
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1.1.1.  Circuit  and  Packet  Switching 

There  are  basically  two  standard  techniques  currently  used  in  communication 
rwtworKs:  line-switching  and  message-switching.  In  line-switching  the  caller  first 
requests  a connection  to  another  subscriber.  After  the  channel  is  established  (usually 
after  the  receiver  has  acknowledged  the  request  and  is  ready  to  communicate) 
information  can  then  be  transmitted  over  the  allocated  channel.  In  message-switching, 
the  sender  starts  transmitting  information  as  soon  as  a path  to  an  intermediate 
switching  node  is  obtained,  thus  allowing  the  use  of  adaptive  routing  techniques. 
Messages  are  temporarily  stored  in  the  switching  nodes  until  completed.  Whenever  a 
path  is  available,  the  message  is  relayed  to  a neighbor  node  along  a path  to  the  final 
receiver.  This  technique  is  also  known,  for  obvious  reasons,  as  Store-and-Forward 
Message-Switching. 

The  choice  of  which  scheme  to  use  depends  on  the  characterstics  of  the 
communication  traffic.  Line-switching  (also  known  as  circuit-switching),  is  more 
advantageous  in  those  situations  in  which  the  transmission  time  for  a message  is  large 
compared  with  the  overhead  needed  to  establish  and  break  down  the  connectioa  It 
has  the  advantage  that  no  buffer  space  is  needed  along  the  intermediate  nodes.  When 
the  size  of  the  messages  decreases,  the  overhead  is  relatively  larger  and  message 
switching  is  preferable. 

To  simplify  error  recovery  and  to  smooth  the  volume  of  traffic  over  the 
rwtwork,  in  certain  message  switching  systems,  messages  are  split  into  fixed  size 
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"packets”  which  are  transmitted  throught  the  network.  In  packet -switched  systems, 
individual  packets  extracted  from  a message  might  travel  different  routes  and  thus 
experiment  different  delays,  necessitating  a reassembly  phase  to  be  performed  at  the 
receiver  end. 

Packet  switched  networks  present  a potential  cost  advantage  for  some  mixes  of 
traffic  over  the  use  of  dedicated  lines  due  to  the  time  sharing  of  the  lines  among  many 
simultaneous  users.  Having  this  type  of  guaranteed  connections  results,  however,  in 
veritable  throughput  and  delay  as  the  load  on  the  network  varies.  This  is  in  contrast 
with  the  behavior  of  a circuit  switched  network:  when  the  network  is  loaded,  the 

'Connection  time  might  increase  (sometimes  to  the  point  that  requests  for  connection 

might  be  rejected)  but  once  a connection  is  established  the  throughput  and  delay 
remains  constant. 

In  a typical  circuit  switching  network  such  as  the  telephone  system,  the  fact  that 
(usually)  only  one  of  the  speakers  is  active  at  a time,  combined  with  the  frequent 
silence  gaps  tends  to  make  inefficient  use  of  the  channel  capacity.  For  the  entire 
network  the  situation  is  even  worse:  most  of  the  lines  are  idle  most  of  the  time  in 

■order  to  provide  the  excess  capacity  that  must  be  available  to  achieve  a specified 

grade  of  service.  Packet  switching  on  the  other  hand,  allows  a greater  flexibility  in  the 
utilization  of  the  network.  The  channels  are  used  only  for  the  time  of  the  actual 
transmission,  the  end-to-end  link  being  provided  by  a logical  channel,  not  a physical 
one. 

1.1.2.  Integrated  Switches 


There  are  three  major  arguments  which  indicate  the  desirability  and  usefulness 
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of  an  Integrated  Communications  System.  These  are:  Cost  reductions,  survivability,  and 
homogeneity  of  service. 

Cost  Reductions 

Use  of  separate  networks  for  data  and  voice  imply  separate  transmission 
facilities,  which  often  travel  parallel  geographical  paths.  The  increased  cost  comes  not 
only  from  the  transmission  equipment  direct  costs  combined  with  the  loss  of  the 
economy  of  scale,  but  also  from  the  increased  number  of  channels  required  to  satisfy 
the  degree  of  service  requirement  for  voice  and  the  maximum  delay  allowed  in 
interactive  transactions.  Communication  networks  are  designed  with  certain  amount  of 
extra  capacity  in  order  to  accommodate  peaks  of  traffic.  The  percentage  of  extra 
capacity  actually  increases  with  smaller  systems.  [Cov75]  shows  that  for  a given 
quality  of  service  (0.02  probability  of  rejection  of  a call),  15  channels  are  required  to 
transmit  9 Erlangs  of  traffic  using  separate  networks  (5  Erlangs  of  Voice  traffic  plus 
the  equivalent  of  4 Erlangs  of  data  traffic),  while  using  an  integrated  network  only  lO'*- 
channels  are  required  without  significant  degradation  of  either  type  of  service.  The 
Key  factor  in  this  saving  is  the  dynamic  allocation  of  the  network  total  bandwidth  to 
the  traffic  demand. 

Survivability 

A second  major  advantage  of  an  integrated  network  is  the  capacity  of  the 
network  to  respond  to  crisis  situations.  Using  separate  systems,  a cripling  accident  will 
reduce  the  capacity  of  the  affected  network  without  being  capable  of  utilizing  any 
spare  capacity  available  in  the  other  network.  An  integrated  system  can  respond 
instantaneously  to  a crisis  by  such  actions  as  allocating  the  surviving  capacity  to  the 
highest  priority  traffic,  preempting  low  priority  calls,  suspending  services  for  low 
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priority  data  packets,  rerouting  circuit  switched  communications,  etc.  Emergency 

adaptabiiity,  as  opposed  to  predictable  allocation  of  traffic  as  a result  of  average  busy  | 

1 

hour  demands,  is  a vital  requirement  in  defense  communications.  j 

Homogeneous  Service  i 

J 

One  of  the  objectives  of  the  integrated  approach  is  to  provide  services  to  the 
largest  possible  community  of  users.  Using  separate  circuit  and  packet  switching 
networks  would  not  allow  interaction  between  subscribers  connected  to  separate  ! 

I 

systems.  Although  it  could  be  argued  that  current  usage  patterns  do  not  require  I 

interaction  between  say,  a telephone  subscriber  and  an  interactive  computer,  future 
applications  can  be  envisioned  in  which  such  interaction  is  desirable  [Cov75].  For 
instance,  we  can  visualize  a query  system  in  which  information  is  stored  in  a computer 

i 

data  bank  and  is  accessible  to  remote  computers  as  well  as  users  connected  to  the 

network  through  a telephone.  Current  research  bn  Speech  Understanding  Systems 

[Red76]  indicates  the  feasibility  of  real  time  man-machine  voice  interactions,  with  task 

restricted  vocabulary  of  a few  hundred  words  but  with  a high  degree  of  accuracy  i 

(better  than  997.  for  single  word  recognition,  slightly  less  accuracy  for  sentences). 

Although  voice  transmission  is  being  used  as  the  canonical  example  of  circuit  j 

switched  traffic,  there  are  other  types  of  real  time  information  transfer,  e.g.:  remote  | 

control  Instrument  monitoring,  graphic  and  television  images,  facsimile,  etc.  all  of  which 
share  the  same  requirements  for  constant  throughput  and  minimal  delay. 
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. 1.2.  Integrated  Switch  Modeling 

1.2.1.  The  SENET  Scheme 

The  integration  of  circuit  and  packet  switching  is  based  on  the  transmission  of 
information  using  a Time  Division  Multiplex  (TDM)  scheme  first  proposed  in  [Cov75].  In 
this  scheme,  time  slices  of  fixed  size  - a frame  period  - are  transmitted  synchronously 
over  high  speed  lines  between  two  adjacent  switches  in  the  network.  The  frame  acts 
as  an  envelope  for  smaller  time  slices  of  variable  size,  each  acting  as  a slot  for 
transmitting  a "packet"  of  information. 

In  this  Slotted  Envelope  NETwork  (SENET)  scheme,  frames  are  transmitted 
synchronously  between  adjacent  nodes  and  a circuit  switching  subnetwork  can  be 
defined  by  the  reservation  of  fixed  slots  inside  a frame.  Since  the  slots  carry  the  same 
amount  of  bits  every  frame  period,  beginning  at  exactly  the  same  point  in  time,  this  is 
equivalent  to  the  reservation  of  dedicated  channels  between  the  two  adjacent  nodes. 
The  rest  of  a frame  can  be  allocated  on  a demand  basis.  If  a suitable  protocol  is 
defined  for  the  proper  interpretation  of  the  information  transmitted  on  this  part  of  a 
frame,  a packet  switching  network  can  be  easily  implemented. 

Traffic  Tvoes 

The  primary  driving  force  in  the  design  of  a communications  system  is  the 
character  of  the  information  it  must  handle.  Based  on  existing  communication 
networks,  three  different  classes  of  traffic  can  be  identified: 

Class  I.-  Characterized  by  long  transactions  requiring  continuous  real  time 
response  (voice,  video,  facsimile).  This  type  of  traffic  can  be 
transmitted  in  the  synchronous  portion  of  the  frame.  They  are 
representative  of  transactions  transmitted  in  a circuit  switching 
"network". 


i 
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i 

Class  II.-  Characterized  by  short  discrete  transactions  requiring  near  real 
time  response  (interactive  data).  This  type  of  traffic  can  be 
transmitted  by  dynamic  allocation  of  slots  in  the  asynchronous 
portion  of  a frame.  They  are  transmitted  in  a packet  switching 
j "network". 

Class  III.-  Characterized  by  long  transactions  requiring  neither  continuous 
I nor  immediate  response  (bulk  data).  This  type  of  traffic  can  be 

I transmitted,  as  the  previous  class,  on  the  asynchronous  portion  of  a 

I frame. 

) 

1 

j The  details  of  a frame  are  shown  in  Figure  1.1.  Starting  at  12  o'clock  a certain 

i number  of  bits  are  reserved  for  CCIS  (Common  Channel  Interswitch  Signaling),  followed 

i 

I by  a Class  I region  containing  the  real  time  traffic.  The  end  of  the  Class  I region  is 

i indicated  in  Figure  1.1  at  5 o’clock.  Class  II  and  III  regions  containing  the  interactive 

i 

I and  bulk  traffic  occupy  the  rest  of  the  frame.  (Unless  we  make  it  explicit,  we  shall 

I make  no  distictions  between  Class  II  and  Class  III  traffic.  We  will  use  the  term  Class  II 

to  indicate  both  interactive  and  bulk  data). 

The  differences  between  the  requirements  and  characteristics  of  the  above 
classification  of  traffic  imply  different  types  of  service: 

Class  I traffic  is  either  accepted  or  rejected,  with  short  connection  delays  and 
without  error  control.  Class  I traffic  requires  low  delay  and  a constant  throughput  in 
order  to  maintain  the  intelligibility  of  the  message.  Class  II  traffic  is  always  accepted 
but  may  incur  a system  delay,  with  short  connection  and  cross-network  delays.  The 
traffic  is  characterized  by  bursts  of  information  followed  by  "waiting"  periods, 
requiring  a high  degree  of  reliability.  Class  III  traffic  is  always  accepted  (although  the 
network  might  temporarily  suspend  this  type  of  service),  with  longer  connection  and 
cross-network  delays  than  the  previous  class.  In  Class  III  traffic,  variations  in  the 
I delay  of  individual  packets  is  not  important  and  this  can  be  used  to  reduce  the  impact 

of  the  long  periods  of  high  traffic  volume.  As  in  Class  II  traffic,  a high  degree  of 
reliability  is  required. 
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The  above  discussion  summarises  the  top  level  characteristics  of  an  Integrated 
Switching  Network.  For  the  purposes  of  the  project,  however,  we  had  to  take  a closer 
look  at  lower  level  considerations  that  will  play  a role  in  the  actual  implementation  of  a 
SENET  system.  The  remainder  of  this  section  outlines  the  major  characteristics  of  a 
possible  implementation, 
j Class  1 Reservation  Tables 

I 

The  real  time  requirements  of  Class  I traffic  dictates  that  it  be  processed  in  a 

• I special  fashion  by  the  integrated  switch.  Since  voice  carries  a significant  amount  of 

I 

redundancy,  a larger  error  rate  can  be  tolerated  in  most  applications  (secure  voice 
transmission  presents  lower  tolerance  to  errors).  By  eliminating  or  reducing  the  need 

! for  error  control,  the  transmission  of  Class  I traffic  can  be  performed  in  a 

! 

straightforward  manner.  The  establishment  of  a logical  circuit  between  two 

I 

I subscribers  is  reflected,  on  each  switch  along  the  path,  in  the  updating  of  Class  1 

I Routing  Tables  internal  to  the  integrated  switches.  These  tables  indicate,  for  each  slot 

in  the  Class  I region  of  an  input  frame,  both  the  output  frame  (output  channel)  and  slot 
position  reserved  for  the  logical  circuit,  as  shown  in  Figure  1.2. 

The  reservation  of  entries  in  the  routing  tables  is  performed  during  the 

{ establishment  of  the  communication  and  remains  in  effect  until  the  communication  is 

! 

I terminated.  The  routing  tables  also  contain  information  about  the  slot  type  (voice, 

I video,  etc.)  and  precedence  (Flash,  Flash  Override,  etc).  These  two  parameters  ere 

I used  to  allocate,  reserve,  and  preempt  the  logical  channels.  The  reservation  of  the 

I . slots  along  the  communication  path  guarantees  a fixed  bandwidth  for  this  type  of 

I 

i traffic.  As  traffic  requirements  vary  during  the  day  the  portion  of  a frame  reserved 

I for  Class  I traffic  can  be  reduced  or  expanded  accordingly. 
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The  input  and  output  links  in  the  network  need  not  necessarily  be  homogeneous. 
For  a given  time  window  length,  say  10  milliseconds,  the  length  of  a frame  varies 
according  to  the  capacity  of  the  link.  Thus,  for  a T1  carrier,  a frame  contains  15,440 
bits.  For  slower  carriers,  the  frame  will  be  accordingly  smaller. 

Allowing  the  use  of  lines  with  different  capacities  permits  the  use  of  similar 
nodes  in  different  capacities  in  the  network.  They  can  appear  as  switching  nodes, 
connecting  high  speed  trunk  lines,  as  part  of  the  access  system  connecting  and 
concentrating  large  numbers  of  low  volume  users,  connecting  HOST  computers  into  a 
communications  network,  etc. 

Asynchronous  Behavior 

The  scheme  does  not  assume  the  existence  of  a master  network  clock.  Each  link 
transfers  frames  at  a constant  rate  (1  frame/time  slice)  but  the  links  are  not 
necessarily  synchronized  among  themselves. 

Requiring  a centralized  network  clock  has  a serious  impact  on  the  reliability  of 
the  network.  In  an  asynchronous  network  each  link  works  independently  from  all 
other  links.  There  is  no  central  critical  component  (a  clock)  whose  failure  will  bring 
the  network  down.  Each  link  between  two  nodes  constitutes  an  isolated  entity  whose 
rate  'Of  failure  has  no  impact  (other  than  perhaps  on  the  volume  of  traffic  permissible) 
on  the  network.  This  approach  also  allows  the  presence  of  transient  errors  in  a link 
which  might  possibly  throw  it  out  of  step.  Link  protocols  and  error  detection  and 
reporting  are,  the  responsibility  of  the  two  nodes  interconnected  by  the  tine. 

Line  Interface  Devices 

The  line  functions  (detection  of  frame  and  packet  headers,  detection  of  speciel 
bit  patterns  or  flags.  Cyclic  Redundancy  Checks,  etc)  are  performed  by  special  purpose 
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hardware  devices.  The  actual  (physical)  input  and  output  operations  are  performed  by 
these  Line  Interface  Devices  (LID)  using  their  Direct  Memory  Access  (DMA)  capabilities. 

The  presence  of  hardware  devices  to  handle  line  error  detection  frees  part  of 
the  node  processing  capability.  From  the  LID,  the  frame  is  loaded  directly  Into  memory 
and  the  event  is  indicated  to  the  integrated  switch  processorfs).  Having  special  line 
handling  devices  allows  the  detection  of  errors  on  a packet  per  packet  basis.  If  error 
detection  were  performed  over  the  entire  frame,  a single  error  anywhere  in  the  frame 
might  make  it  invalid  and  therefore  the  frame  might  have  to  be  retransmitted,  an 
unacceptable  situation  from  the  point  of  view  of  Class  I transmission. 

Bvtes  as  Transmission  Units 

We  assume  an  8-bit  byte  as  the  smallest  unit  of  information  transmission.  Thus, 
, the  integrated  switch  processors  will  treat  a frame  as  a vector  of  8-bit  characters. 
Headers,  packets,  and  frames  are  therefore  assumed  to  contain  an  integral  number  of 
characters. 

Although  the  actual  transmission  of  information  is  done  on  a bit  serial  basis, 
existing  processors  rarely  use  the  bit  as  the  unit  of  directly  addressable  information. 
A bit  addressing  capability  would  imply  a larger  address  field  in  the  instructions,  a 
serious  penalty  to  pay  for  a rarely  needed  capability.  This  is  not  an  unreasonable 
assumption.  Packet  sizes  are  typically  several  hundred  bits  long  and  the  possible 
waste  (if  any)  will  be  a small  fraction  of  the  traffic. 

Packet  Transmission  Format 

The  use  of  the  Advanced  Data  Communication  Control  Procedures  (ADCCP) 
[ADC75],  or  variant  thereof,  seems  reasonably  well-suited  to  this  application.  Each 
packet  to  be  transmitted  is  augmented,  as  in  ADCCP,  by  hardware-generated  header 
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j end  trailer  fields.  The  header  field  identifies  the  start  of  a packet  (by  a unique  flag- 

i ■ character)  and  passes  control  information  to  the  Line  Interface  Device  (LID)  at  the 

I 

I destination.  The  trailer  field  contains  the  cyclic  redundancy  field  followed  by  the  flag- 

' character  to  indicate  the  end  of  the  packet.  By  so  delimiting  the  packet  with  unique 

! 

flag-characters,  error  detection  becomes  straightforward. 

t 

I Figure  1.3  illustrates  a possible  transmission  format  along  the  lines  of  the  above 

I discussion.  The  figure  shows  the  flag-characters  being  used  as  idle  characters 

I 

indicating  an  idle  transmission  line.  Other  conventions  are  possible.  By  use  of  a bit- 
stuffing technique  the  flag -character  can  be  guaranteed  not  to  occur  within  the 
augmented  packet.  The  reverse  operation  at  the  destination  recreates  the  original 
! packet. 

I • 

I The  design  thus  described  is  self-synchronizing  in  the  sense  that  there  is  no 

I 

i need  for  explicit  synchronizing  signals  or  procedures.  The  source  node  can  send  a 

packet  anywhere  inside  the  asynchronous  portion  of  a frame.  Similarly,  the  destination 
detects  the  start  of  a transmission  simply  by  noticing  the  absence  of  a flag-character. 
' Because  of  this  independence  between  source  and  destination  ends  of  the  lines,  only 

I 

I local  clocks  at  the  source  ends  are  necessary  to  time  the  frames. 

f 

Class  I Transmission 

I 

Since  Class  I slots  are  small  (say,  80  bits  for  an  8 Kbps  vocoder  channel  [see 
For 75]),  sending  each  as  a separate  packet  would  entail  a great  deal  of  overhead. 
Indeed,  the  main  reason  for  using  separate  packets  for  Class  II  traffic  is  error 
detection,  which  is  not  a concern  with  Class  I transmissions.  The  reasonable  conclusion 
is  to  send  the  entire  Class  I portion  of  each  frame  as  a single  packet.  This  has  the 

i 

advantage  of  minimizing  the  number  of  overhead  bits  required  (a  singla  header /trailer 

I 

I 
I 
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pair).  However,  It  is  important  that  this  Class  I pacKet  should  always  be  accepted  by 
the  destination  LIO,  even  if  a transmission  error  is  detected.  What  is  needed  is  a tvoe 
field  in  each  packet  header  which  identifies  the  packet  as  either  Class  I or  Class  IL  (In 
a modified  AOCCP,  the  "control”  field  of  Figure  1.3  could  be  used  for  this.)  The  solution 
to  the  problem  is  then  to  have  the  destination  LID  intentionally  ignore  any  possible 
error  indication  for  Class  I packets.  As  a consequence,  control  information  used  to 
establish  or  break  Class  1 communications  must  be  send  as  Class  II  packets.  Thus,  a 
frame  can  not  be  dedicated  exclusively  to  Class  I traffic,  some  smalt  partion  must  be 
reserved  for  Class  II  control  packets  for  this  purpose. 

Frame  Generation  and  Timing 

According  to  the  original  Coviello  and  Vena  concept,  new  real-time  data  appears 
say,  every  10  milliseconds  (in  a T1  carrier  this  corresponds  to  15,440  bits/frame). 
Hence,  every  10  milliseconds  a Class  I packet  must  be  transmitted.  The  start  of  a 
frame  would  simply  be  indicated  by  receipt  of  each  Class  I packet  header.  No  explicit 
start-of-frame  marker  would  be  required.  Whenever  a new  Class  I packet  begins  to 
arrive,  the  LIO  can  interrupt  its  attached  switch  processor  to  announce  receipt  of  the 
previous  frame.  In  Figure  1.4  for  example,  a schematic  "snapshot"  of  packets 
traversing  a transmission  from  right  to  left,  the  destination  LID  would  be  storing  frame 
1 into  a memory  buffer  (via  DMA)  as  it  is  arriving.  When  packet  I2  begins  to  arrive, 
the  LID  would  signal  an  interrupt  to  indicate  receipt  of  frame  1.  When  I3  begins  to 
arrive,  receipt  of  frame  2 would  be  signalled,  and  so  on.  In  other  words,  a frame 
consists  of  a single  Class  I packet  followed  by  zero  or  more  Class  II  packets.  The 
intervals  between  Class  I packets  are  filled,  either  partially  or  completely,  by  Class  II 
packets. 
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1.2.2.  Task  Decomposition 

• 

Before  going  into  the  details  of  the  functional  decomposition  of  the  system  we 
must  provide  an  overview  of  C.mmp  and  its  architecture  since  it  plays  a central  role  in 
many  of  our  design  decisions. 

C.mmp  (Figure  1.5)  is  a multiprocessor  system  designed  and  built  at  CMU  under 
the  sponsorship  of  the  Advanced  Research  Projects  Agency  (ARPA)  of  the  Department 
of  Defense.  The  system  consists  of  (up  to)  16  miniprocessors  (DEC  PDP- 11/20  and 
PDP-ll/AO)  connected  through  a central  crosspoint  switch  to  a large  shared  memory 
(up  to  32  million  bytes).  In  addition,  each  processor  has  a small  amount  of  local  storage 
for  private  code  and  data.  The  use  of  a large  shared  memory  makes  it  possible  to 
define  multiprocessor  sub-systems  by  the  proper  allocation  of  the  storage  to  small 
clusters  of  cooperating  processors.  Both  infer-  and  intra-cluster  communication  and 
synchronization  can  be  implemented  by  sharing  buffers  and  interlocks  acessable  by 
the  individual  processors.  Although  C.mmp  also  provides  a limited  interprocessors 
interrupt  facility,  it  was  not  used  in  our  system  for  reasons  explained  in  Section  3.3. 

The  emulation  of  a single  integrated  switch  is  performed  by  a cluster  of 
processors  sharing  both  code  and  storage.  Other  processors,  external  to  the  cluster 
are  used  to  simulate  the  outside  world  (the  network)  and  are  in  charge  of  feeding 
multiple  frame  streams  (simulating  multiple,  asynchronous,  input  channels)  and 
collecting  statistics  about  the  behavior  of  the  integrated  switch. 

Inner  and  Outer  Loop  Functions 

The  functions  to  be  performed  in  an  Integrated  Communications  Network  can 
best  be  studied  if  we  recognize  a classification  in  terms  of  their  impact  on  the  global 
network  behavior  and  in  terms  of  their  processing  requirements  on  the  individual 
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switches.  The  following  decomposition  separates  those  functions  that  are  performed 
by  the  integrated  switches  in  order  to  establish  and  maintain  communication  with  the 
other  switches  of  the  network  (the  relay  functions)  from  those  that  are  performed  in 
order  to  establish  and  maintain  the  accessability  to  the  network  by  the  local 
subscribers  (the  regional  functions).  The  classification  can  be  further  refined  by 

a 

identifying  those  functions  that  are  continuosly  performed  as  part  of  the 
communications  task  from  those  that  are  performed  infrequently,  as  a result  of 
exceptional  conditions  (e.g.,  error  detection  and  recovery)  or  during  the 
initialization/termination  phases  of  a communication  between  two  subscribers. 

Functions  marked  with  an  asterisk  (*)  are  "outer-loop*  functions,  i.e.,  functions 
which  require  negligible  execution  time  compared  to  the  "inner-loop"  functions  (those 
not  marked  with  an  asterisk).  Only  "inner-loop"  functions  are  included  in  the  first 
version  of  the  system. 

Relay  Functions  (Trunk  lines) 

a 

(1)  Routing  Selection. 

- Choice  of  output  line  to  send  message  out  on,  including  messages 
originating  at  this  node. 

- Essentially  table-driven  at  this  level. 

(2)  Message  Acknowledgement. 

- Acknowledging  received  messages. 

- Timeout  on  expected  acknowledgements  for  sent  messages. 

(3)  Channel  Discipline. 

- Frame  decomposition  and  assembly. 

- Allocating/compacting  real-time  message  slots  in  frames. 

(4a)  MaintainarKe  of  Routing  Tables. 

-Timing  of  message  delays  to  neighboring  nodes  for  adaptive  routing. 


F 


i 
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(5*)  Hardware/Software  Fault  Detection. 

- Intra-node  detection:  Timeout,  NXM,  data  structure  integrity  check, 
reliability  analysis. 

- Inter-node  detection/correction:  Status  of  other  nodes  as  viewed  over 
interconnection  links;  external  load/restart  capabilities. 

Regional  Functions  (Access  lines) 

(1)  Anaiysis/Validation/Generation  of  Packet  Header  Fields.  ^ 

- Precedence  check:  is  this  terminal  authorized  to  use  this  level  of 
precedence? 

- Security  authorization  check:  is  this  terminal  authorized  to  send/receive 
messages  of  this  security  level? 

- Comparison  of  header  security  keys:  do  they  match? 

- Segment  count:  check  for  any  missing  or  out-of-sequence  and 
appropriate  actions  if  so. 

- Generation  of  any  necessary  header  information  for  originating 
messages. 

(2)  Flow  Control  Functions. 

- Access  denial:  refusing  originating  traffic  for  categories  not  being 
accepted  by  destination  (e.g.,  because  of  congestion) 

- ()ueue-fength  monitoring,  to  detect  congestion. 

- Purging  of  low  category /precedence  traffic  from  queues  if  necessary. 

- Delay  of  acknowledgements  if  congested. 

(3)  Logical  Channel  Control. 

- Opening/closing  of  logical  channels,  especially  for  real-time  connections. 

(4)  Precedence  Handling  Functions. 

- For  high  precedence  messages. 

(5*)  Identification/Validation  of  Terminals. 

- Table  lookup,  possible  password  check,  etc. 

(6a)  Compatibility  Conversions. 
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- Character  translation,  mode/format  conversions. 

(7*)  Maintainance  of  Regional  Tables. 

- Line  tables,  terminal  ID  tables,  logical  channel  tables,  etc. 

(8*)  Hardware/Software  Fault  Detectioa 

(9*)  Statistics  Gathering/Reporting. 

The  scope  of  the  first  year  project  centers  around  the  inner  loop  functions  and 
their  emulation  on  C.mmp.  Thus  we  only  deal  with  an  isolated  node  although  the 
presence  of  the  "outside  world"  (i.e.  network)  is  a driving  force.  This  will  be 
elaborated  later,  when  we  discuss  the  Script  Generator  and  the  Statistics  Collection. 
Software  Organization 

The  handling  of  real  time  and  high  priority  traffic  indicated  that  a conventional 
multiprogramming  approach  could  not  perform  the  mission.  The  time  required  to  switch 
contexts  (in  the  order  of  several  milliseconds  in  Hydra)  could  very  well  prevent  a 
processor  from  fulfilling  a request  within  an  acceptable  time.  However,  the  use  of 
homogeneous  processors  and  shared  storage  permited  us  to  implement  the  system  as  a 
set  of  procedures  directly  executable  by  any  processor,  operating  on  shared  data. 
Thus,  as  in  multiprocessing,  no  processor  is  dedicated  to  any  particular  task.  Context 
switching  is  however,  replaced  by  a faster  task  switching  scheme  described  in  the  next 
paragraph. 

The  scheduling  of  the  tasks  is  performed  via  a set  of  priority  ordered  queues 
(Figure  1.6).  Some  queues  are  dedicated  to  a specific  task  (for  instance,  routing). 
Others  are  dedicated  to  families  of  related  tasks.  In  the  idle  state,  processors  are 
continuously  executing  the  scheduling  task,  namely,  scanning  the  queues  according  to 
their  priorities.  When  a non-empty  queue  is  found,  the  scheduling  task  takes  the  first 
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element  of  the  queue  and  the  processor  executes  ("calls'*)  a procedure  that  implements 
the  tasK  that  was  found  in  the  scheduling  queue.  The  execution  of  a task’s  procedure 
might  result  in  the  creation  and  queueing  of  other  task  requests.  When  the  task’s 
procedure  is  completed,  the  processor  resumes  the  execution  of  the  main  loop,  the 
scheduling  task. 

Class  1 Traffic  Processing 

As  indicated  before,  voice  and  other  real  time  traffic  carries  a significant  amount 
of  redundancy  and  a larger  error  rate  than  on  data  can  be  tolerated.  This  suggests 
that  performing  error-checking  and  generating  acknowledgements  for  Class  I packets 
can  be  avoided.  However,  the  establishment  and  maintainance  of  a Class  I channel  is  a 
critical  requirement  thus,  the  control  messages  interchanged  between  network  nodes  in 
order  to  establish,  re-route,  and  break-down  Class  I connections  must  be  thoroughly 
reliable.  These  type  of  messages  are  best  transmitted  as  Class  II  packets.  It  should  be 
noticed  that  given  the  long  holding  time  for  a typical  voice  channel  (180  seconds  or 
18,000  frames  in  a lOms/frame  network),  the  few  messages  that  might  be  needed  for 
control  purposes  represent  a negligible  amount  of  extra  traffic. 

When  a frame  arrival  is  detected  by  the  input  LID,  an  event  notice  is  posted  in  a 
pre-assigned  location.  As  soon  as  a processor  becomes  available,  after  completion  of 
its  current  task,  the  arrival  notice  is  removed  and  the  processor  initiates  the  task  of 
decomposing  the  frame.  This  task  gives  special  consideration  to  Class  I traffic,  i.e.  to 
minimize  the  switch  delay,  transmission  of  Class  I traffic  is  given  precedence  over  most 
Class  II  related  tasks. 

The  handling  of  incoming  Class  I traffic  by  the  switch  program  at  a trunk  node  is 
basically  a "scatter-write”  operation,  where  the  Class  I slots  arrive  in  a contiguous 
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buffer  but  are  then  copied  piecemeai  to  various  output  buffer  iocations,  as  determined 
by  the  Ciass  I Routing  Tabies. 

Irnlependent  of  the  "scatter-write”  implementation,  the  primary  design  goal  is  to 
minimize  the  delay  time  through  a node.  However,  there  are  some  pitfalls  that  one 
must  be  aware  of  in  constructing  such  a design.  These  involve  the  possibility  of 
slippage  or  overwrite  of  Class  I slots.  The  former  problem  consists  of  the  introduction 
of  random  "silence"  gaps  when  a Class  I slot  misses  its  output  frame  and  is  delayed 
one  extra  frame  period  in  the  switch.  The  latter  problem  occurs  when  a Class  I slot  is 
overwritten  by  the  slot  arriving  in  the  next,  frame  and  before  the  output  frame  has 
been  transmitted.  These  two  problems  are  studied  in  more  detail  in  Appendix  I. 

Class  11/111  Traffic  Processing 

During  frame  decomposition,  the  individual  Ciass  II  packets  are  isolated,  and 
pointers  to  each  individual  packet  are  posted  in  a Class  II  copy  task  queue.  When  this 
frame  decomposition  is  terminated,  the  processor  returns  to  the  available  pool  and 
competes  with  other  idle  processors  for  pending  tasks. 

Figure  1.7  depicts  the  sequence  of  tasks  in  the  integrated  switch  system.  Qq  is 
the  Input  Frame  Queue,  Qj  is  the  Class  I Copy  Queue,  and  Q2  is  the  Class  II  Copy 
Queue.  The  Class  II  copy  tasks  consist  of  copying  the  data  packets  into  individual 
buffers  (due  to  higher  reliability  requirements,  data  packets  must  be  kept  in  the  switch 
after  they  are  retransmitted  while  waiting  for  an  acknowledgement).  After  a packet 
has  been  stored  in  a buffer,  different  tasks  might  be  created  depending  on  the  nature 
of  its  type,  precedence,  and  final  destination  (Q3  and  Q^).  Some  Class  II  packets  may 
convey  line  control  information  or  acknowledgements.  These  packets  will  be  either 
forwarded  to  other  switches  or  a specific  control  function  will  be  performed  and  the 
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pacKet  will  then  be  deleted.  For  data  packets  with  a foreign  final  destination,  routing 
and  acknowledgement  generation  tasks  are  created.  Packets  addressed  to  a local  HOST 
might  instead  be  placed  into  a re-assembly  queue.  The  routing  tasks  are  implemented 
on  Q5  and  O5. 

Queueing  Models  (An  Introduction) 

Although  the  theoretical  performance  model  of  the  SENET  implementation  on 
C.mmp  is  fully  elaborated  in  Chapter  2,  it  is  appropriate  to  make  a few  remarks  about 
the  queueing  characteristics  of  the  integrated  switch  as  they  relate  to  the  preceeding 
discussion.  • a 

Class  I traffic  is  a fuliy  deterministic  process.  The  combination  of  a relatively 
long  holding  time  (several  thousand  frames)  together  with  a constant  service  request 

j (essentially  looking  up  the  Class  I Routing  Tables  and  copying  the  Class  I slot  into  the 

» 

! 

I appropriate  output  frame)  allows  us  to  simplify  the  modeling  of  this  part  of  the  system, 

j Class  II  packets  present  a totally  different  picture.  The  need  for  error  checking, 

I generation  of  acknowledgements,  precedence  driven  processing,  computation  of  routing 

information,  management  of  buffer  space,  etc.,  result  in  a truly  random  service  request 

I 

j for  each  packet  that  arrives  at  the  integrated  switch. 

' The  combination  of  Class  1 and  II  traffic  results  in  a system  in  which  the  inter- 

I 

: arrival  time  of  the  request  for  service  on  the  different  queues  in  the  system  is  neither 

^ exponentially  distributed  (a  common  assumption  in  theoretical  studies)  nor 

deterministic.  For  each  frame  there  is  a constant  level  of  Class  I traffic  requests  and 
I some  random  level  of  Class  II  traffic  requests.  Although  there  are  no  simple  models  to 

I 

I analyse  such  a system,  several  approximations  were  developed  at  CMU  and  are 

I 

I presented  in  Chapter  2. 
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1.3.  Experimentation  Utilizing  the  Physical  Model 


1.3.1.  Experimental  Design 

The  general  design  of  the  experiment  on  C.mmp  uses  two  processors  outside  of 
the  switching  node  for  both  the  generation  of  the  script  and  the  analysis  of  the 
results.  The  script  generator  process  (described  below)  generates  the  incoming 
frames  for  each  of  the  channels  and  stores  them  in  a set  of  buffers  reserved  for  this 
purpose.  Currently,  there  is  a ring  of  eight  buffers  on  each  channel  to  allow  for 
variation  in  the  speed  of  the  script  generator  process  (a  frame  with  few  packets  takes 
less  time  to  generate  than  one  one  with  many  packets). 

The  analysis  process  is  also  outside  of  the  simulation  per  se.  It’s  task  is  to 
examine  all  the  Class  I slots  and  all  the  packets  send  on  each  frame,  for  each  output 
channel.  Delays  of  packets  and  Class  I slot  transfers  are  computed  for  later  calculation 
of  mean  and  standard  deviation  of  various  categories  of  packets. 

In  addition  to  the  Script  Generator  process  and  Analysis  process,  there  is  one 
other  component  of  the  simulation:  a KWP-ll  programmable  timer  which  is  supported 
under  HYDRA  to  provide  precise  timings  of  frame  durations,  arrival  times,  etc.  Because 
of  the  limitations  of  running  under  the  HYDRA  operating  system,  as  explained  in  Section 
3.3,  the  simulation  program  is  not  allowed  to  field  the  timing  interrupts  directly. 
Instead,  a form  of  polling  is  used,  where  each  of  the  "slave"  processors  (the  ones 
carrying  out  the  simulation)  frequently  check  to  see  if  an  interrupt  has  been  signalled. 
Given  a reasonable  number  of  slave  processors,  the  delay  before  an  interrupt  signal  is 
recognized  can  be  made  reasonably  small. 
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The  overall  configuration  can  be  symbolically  described  as  "1+1+N"  where  "N" 
represents  the  N slave  processors  in  the  simulation,  an^"!-*-!”  represents  the  Script 
Generator  and  Analysis  processors. 

The  original  thought  in  the  simulation  was  to  pre-generate  a large  script  and 
simply  read  it  in  as  it  was  required.  However,  since  secondary  memory  would  be 
required  for  this,  there  was  no  satisfactory  way  to  guarantee  that  (a)  segments  of  the 
script  could  be  read  into  main  memory  in  time,  and  (b)  the  operating  system  would  not 
usurp  some  of  the  processors  for  large  periods  of  time  (tens  of  milliseconds)  to 
perform  the  input/output  operations.  Hence,  it  was  necessary  to  ensure  that  the 
simulation  would  be  entirely  compute-bound  and  not  do  any  input/output  during  an 
actual  run.  This  required  that  the  script  and  analysis  be  performed  " on-the-fly".  The 
main  drawback  with  this  approach  is  the  limitations  of  how  fast  the  script  can  be 
generated,  and  how  fast  the  results  can  be  analyzed  and  accumulated.  More  will  be 
said  on  this  later. 

The  Script  Driver 

A separate  processor  is  dedicated  to  simulating  the  traffic  to  the  node.  This 
processor  continously  runs  the  Script  driver  program  generating  the  traffic  according 
to  specifications  told  in  a dialogue  during  the  initial  set  up.  The  system  timings  are 
kept  with  a programmable  hardware  clock  in  conjunction  with  a fast  interrupt  routine 
which  sets  soflw.-'re  interrupts,  indicating  arrival  and  departure  of  the  frames  or  other 
•vents.  These  interrupts  are  attended  to  by  system  processors  hunting  for  work.  In 
order  to  irieet  the  randomly  varying  amount  of  required  work,  the  driver  does  not 
work  on  a frame  timing  basis.  Instead,  for  each  line,  there  is  a ring  of  eight  buffers  and 
c variable  pointing  to  the  current  frame  , and  the  script  driver  (after  taking  a 
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headstart)  continuously  fills  empty  buffers  with  new  frames.  Each  frame  carries  a 
sequence  number  which  apart  from  identifying  frames  is  also  used  for  finding  its 
arrival/departure  times,  since  these  occur  at  a fixed  specified  interval.  The  actual 
message  is  of  no  concern  to  the  node,  and  no  attempt  is  made  to  generate  ’garbage’ 
messages  in  order  to  conserve  space  and  time.  Only  headers  are  processed  and 
generated.  The  format  of  frame  and  headers  is  described  in  Figure  1.8. 

Node /Network  Parameters.-  The  parameters  that  can  be  specified  for  node  and  the 
traffic  nature  are  as  follows: 

Number  of  lines  connected  to  the  node.  A "soft"  limit  of  4 channels  is 
assumed,  mainly  for  buffer  allocation  purposes.  If  during  the 
experiment  runs  need  is  felt  for  having  a bigger  number,  it  can  be 
easily  accomplished. 

Frame  duration  (typically  10  msec.),  relative  arrival  and  departure  times  of 
frames  on  every  line.  This  information  is  used  for  initialising  and 
directing  the  hardware  clock  routine  as  to  when  and  which  interrupts 
to  post. 

Line  Parameters.-  Each  line  is  characterised  by: 

Speed  in  Kilobauds. 

Composition  of  the  real  time  packets  (i.e.  length  and  destination  of  all  real 
time  slots).  In  this  set  of  experiments  the  duration  of  the  run  is  kept 
much  smaller  then  typical  holding  times  of  real  time  traffic.  Hence 
the  reservations  for  these  are  preset  and  are  not  changed  during  the 
run. 

Composition  of  the  Class  II  and  III  traffic  region.  There  are  twenty  one 
levels  of  priorities  in  Autodin-II  specs,  however  such  a large  number 
of  levels  are  of  no  interest  for  experiments,  since  they  make  design 
and  analysis  of  experiments  unnecessarily  difficult.  Therefore  only 
six  classes  are  implemented,  and  during  analysis  the  larger  set  can 
be  meaningfully  mapped  onto  them.  The  six  classes  are  Control, 
Realtime,  Data  High,  Data  Low,  Bulk  High,  Bulk  Low.  The  fractions 
indicated  are  used  as  discrete  linear  distribution  for  determining  type 
for  a packet. 

Direction  of  traffic  flow.  That  is  destination  of  packets  incoming  on  this 
line.  Fractions  are  indicated  for  every  other  line,  which  are  used  as 
discrete  linear  distribution  for  deciding  the  destination  of  packet  at 
random. 
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Average  number  of  data  packets  per  frame  and  their  distribution.  A 
poisson  distribution  is  a good  choice  for  this. 

Length  of  packets  and  their  distribution 

I 

Additional  Parameters.-  Apart  from  these  above  line  characteristics  the  following  also 
need  to  be  specified: 

How  long  the  system  should  run  . 

How  long  the  system  should  run  before  steady  state  is  obtained  and 
statistics  collection  can  be  started. 

Times  for  statistics  collection  relative  to  frames  arrival  timt'S. 

Seeds  for  the  random  number  generator. 

The  script  driver  prompts  for  the  above  information  in  the  beginning.  During  this 
phase  the  script  driver  prompts  the  user  for  the  emulation  parameters  and  allows  the 
user  to  set,  display,  and  modifify  parameters  specified  for  a previous  experiment.  An 
example  of  a work  session  in  Appendix  II  illustrates  its  exact  use  and  is  self 
explanatory.  A consistency  check  is  made  as  far  as  possible.  In  case  of  meaningless 
specs,  error  messages  are  issued  and  for  doubtful  cases  warnings  are  given.  Thus 
specifying  minimum  length  of  a packet  greater  than  its  maximum  length  would  cause  an 
error  message  and  cause  the  question  to  be  repeated.  Specifying  a large  average 
number  of  packets  on  a slow  line  would  solicit  a warning.  A run  time  record  of  the 
nature  of  traffic  actually  generated  is  kept  and  later  the  statistics  in  terms  of  the 
traffic  intensity,  average  number  of  packets/frame,  average  length  of  packets,  fraction 
of  traffic  in  various  classes,  etc.  is  reported. 

Operation  gf  ihg  Experiment 

The  experimental  results  are  obtained  from  series  of  discrete  "runs*.  Each  run 

e 

represents  a completely  independent  simulations  the  input  parameters  can  be  similar  to 
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those  of  other  runs,  or  widely  different.  The  experimenter  can  vary  the  parameters  to 
create  a particular  situation  worthy  of  study.  The  process  is  interactive:  the 
experimenter  can  observe  the  results  of  one  run  and  immediately  apply  that 
information  to  the  next  run. 

The  cycle  of  operation  in  running  the  simulation  is  as  follows.  Initially,  the 
experimenter  must  interact  with  HYDRA  to  create  the  necessary  processes,  I/O 
connections,  etc.  This  is  not  repeated  during  the  remainder  of  the  session.  Once  the 
initial  set-up  and  allocation  has  been  done,  the  program  asKs  for  input.  Appendix  II 
(on  the  Script  Generation  parameters)  shows  an  example  of  a dialogue  with  the  input- 
section  of  the  program.  It  is  not  necessary  to  always  type  in  a whole  net  set  of  data 
each  time  — if  the  parameters  are  similar  to  the  previous  run,  the  input-routine  allows 
one  to  only  specify  items  that  need  to  be  changed. 

After  input,  the  experimenter  allows  the  program  to  begin,  and  waits  until  it 
returns  to  the  terminal  to  asK  where  the  output  should  be  sent  (eg,  line  printer, 
terminal,  etc.).  After  output  is  completed,  the  program  is  ready  to  accept  new  data  (or 
changes  to  the  last  data)  and  run  again. 

Explanation  of  Output 

Output  from  the  program  is  given  in  the  form  of  tables,  which  require  some 
explanation.  The  first  tables  to  be  printed  concern  the  Script  Generator  process.  It  is 
useful  to  know  exactly  the  attributes  of  the  script  that  was  generated.  For  one  thing, 
this  serves  as  a check  that  the  Script  Generator  is  performing  correctly  and 
generating  data  that  corresponds  to  the  parameters  read  in.  In  addition,  measurement 
of  the  generated  script  is  necessary  to  obtain  some  of  the  parameters  which  cannot  bo 
specified  directly  as  input.  (For  example,  average  data  capacity  used  cannot  be 
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specified  directly  as  a distribution  parameter,  since  it  also  depends  on  how  often  a 
pacKet  must  be  delayed  until  a subsequent  frame  because  of  lack  of  room  --  this 
cannot  be  calculated  ahead  of  time  easily.) 

Figure  1.9  shows  the  first  page  of  program  output.  The  first  section  simply 
displays  the  input  parameters.  This  is  followed  by  the  actual  measurements  on  the 
script:  average  number  of  packets  sent  on  each  input  channel,  and  fraction  send  to 
each  of  the  output  channels;  average  packet  length  (in  bytes),  fraction  of  data  capacity 
used  on  each  input  channel,  and  total  frame  capacity  used  (including  Class  I traffic). 
Finally,  the  breakdown  of  the  data  packets  sent  by  priority  is  displayed. 

Figure  1.10  illustrates  the  output  from  the  Analysis  process  for  a sample  run. 
The  first  table,  entitled  ”Realtime  Delays"  (ie,  Class  I traffic)  is  basically  a histogram 
showing  the  total  number  of  Class  I slots  transferred  within  n frames.  It  is  clearly 
desirable  to  have  the  cross-node  delay  for  Class  I traffic  be  as  small  as  possible,  but 
there  are  dangers  associated  with  trying  to  make  the  delay  too  short  (see  Appendix  I 
on  Class  I slot  slippage/overwrite).  In  the  simulation,  three  buffers  are  used  for  the 
Class  I traffic.  This  is  necessary  to  avoid  the  possible  dangers  mentioned,  but  it  means 
that  the  cross-node  delay  will  always  be  a duration  of  two  frames.  In  addition,  any 
relative  skew  between  incoming  frame  and  outgoing  frame  is  added  onto  the  two 
frames’  delay.  Hence,  all  the  slots  from  a specified  channel  should  be  transferred  in 
exactly  two  frame’s  delay  (plus  skew,  which  can  range  from  0 up  to  almost  a frame  in 
duration,  depending  on  the  relative  frame  arrival  time  compared  to  the  frame 
departure  time).  Class  I slots  transferred  during  any  other  frame  indicate  an  error  or 
exceptional  condition,  such  as  the  node  program  being  unable  to  meet  the  deadlirte 
requirement.  Similarly,  the  "Empty  Slots”  column  indicates  the  number  of  empty  Class  I 
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slots  seen  by  the  Analysis  processor.  This  is  another  indication  that  the  simulation 
processors  were  unable  to  meet  their  deadline  requirements  for  Class  I processing. 

The  second  table  is  a bit  more  self-expianatory.  Each  Class  II/III  packet 
receives  a time-stamp  upon  entering  the  system  from  the  Script  Generator.  The 
Analysis  process,  when  it  sees  each  packet  at  the  output  end,  computes  the  cross- 
node delay  for  that  packet,  and  keeps  a running  sum  of  the  delays  and  squares  of  the 
delays  (for  calculation  of  standard  deviation).  The  results  are  tabulated  by  priority.  In 
addition  to  the  standard  statistics,  an  attempt  is  made  to  calculate  the  buffer-lifetimes 
used  by  the  data  packets  (ie,  packet  lengthsdelay).  This  lifetime  does  not  currently 
include  any  estimation  of  the  time  required  for  a return  ackowledgement  (which  is  a 
major  factor). 

A number  of  checks  have  been  included  in  the  program  to  test  whether  the  real- 
time deadlines  are  being  met.  Whenever  critical  tasks  are  not  performed  in  time, 
certain  counters  are  incremented.  These  are  displayed  just  before  the  delay  tables, 
and  have  the  following  meanings.  "Noverrun"  gives  the  number  of  times  that  the  Script 
Generator  was  unable  to  keep  up  with  demand  for  incoming  frames.  The  major  factor 
here  is  the  number  of  packets  to  be  created  in  the  frame,  and  thus  how  many  times 
the  random  number  generator  must  be  invoked.  "Vstattooslow”  represents  the  number 
of  times  that  the  Analysis  routine  had  to  give  up  on  analysis  of  the  voice  (ie.  Class  I) 
data  because  the  buffer  had  just  become  the  current  buffer  being  loaded. 
"OPoverrun"  is  the  number  of  times  that  the  available  space  for  outgoing  packets  was 
completely  filled.  Currently,  there  are  a total  of  six  XCVs”  (Channel  Command 
Vectors)  which  serve  as  a ring  of  buffers,  much  as  the  Class  I buffers  are  used.  Each 
outgoing  frame  consists  of  the  appropiate  Class  I buffer  followed  by  the  list  of  data 
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packets  on  the  appropiate  CCV.  When  a packet  is  to  be  sent  and  all  the  CCVs  are  full, 
then,  DPoverrun  is  incremented.  Finally,  "VCoverrun"  indicates  the  number  of  times 
that  the  voice,  or  Class  I,  copy  task  was  unable  to  meet  its  deadline  before  the 
specified  buffer  began  to  be  sent  out. 

Figure  1.11  shows  the  remainder  of  the  output  from  a run.  The  first  section 
shows  the  breakdown  of  tasks  performed  by  each  of  the  "slave"  processes.  The  tasks 
are  shown  across  the  top,  and  the  number  of  times  each  process  looked  at  that  task 
queue,  and  the  number  of  times  that  that  process  found  a task  to  do,  are  tabulated. 
The  ratio  of  the  two  is  also  calculated  and  printed. 

The  second  section  gives  average  queue  lengths  in  the  system,  sampled  on  a 
regular  basis  (once  per  frame-duration).  An  idea  of  the  amount  of  interference 
between  the  processes  is  obtained  by  examining  how  often  a queue  was  locked  by  a 
process  when  access  was  attempted  by  another  process.  Basically,  (he  tasks  table  and 
the  queue-length  table  are  not  directly  related  to  the  performance  of  the  node 
simulation,  but  they  can  offer  information  on  the  internal  workings  of  the  simulation. 

1.3.2.  Experimental  Results 

Data  Traffic 

System  Parameters: 

Line  Speed:  1.544  megabits/sec.  (T1  carrier) 

Frame  Length:  10  milliseconds 
Number  of  Channels:  2 (full  duplex) 

Number  of  Processors:  3 (POP-1 1/20) 

Frame  Departure  Time:  0 msec. 

Channel  0 Frame  Arrival:  1 msec. 

^ Chanrtel  1 Frame  Arrival:  9 msec. 

I 

I 

I 
I 


I 
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In  the  first  series  of  experiments  we  studied  the  limitations  of  a small  switching 
node  (3  miniprocessors).  The  experiments  were  designed  to  compute  the  average 
intranode  pacKet  delay  in  the  absence  of  real  time  traffic.  I.e.  we  were  measuring  the 
ability  of  the  configuration  to  respond  to  various  degrees  of  traffic  intensity  and  the 
speed  with  which  Class  II  and  III  packets  could  be  transmitted  between  two  incoming 

a 

and  two  outgoing  lines.  The  traffic  was  assumed  to  consists  of  two  types  of  packets, 
each  acting  as  representative  of  Classes  II  and  III  respectively.  The  relative  volumes 
of  each  traffic  type  were  set  at  207.  (Class  II)  and  807.  (Class  III).  Moreover,  the  packet 
lengths  were  drawn  from  the  same  uniform  distribution,  bounded  beween  200  and 
2200  bits  respectively 

It  was  observed  in  preliminary  runs  that  one  of  the  limitations  of  the  system 
would  be  the  ability  of  the  C.mmp  processors  to  move  the  data  packets  from  an 
incoming  frame  buffer  into  the  individual  packet  buffers.  If  we  consider  that  a PDP- 
11 /20  takes  in  the  order  of  5 microseconds  to  perform  a memory-memory  word 
movement,  just  copying  the  information  contained  in  a full  frame  would  take: 
15440bits/frame*l word/1 6bits*5microseconds/word  - 4825microseconds.  In  other 
words,  almost  half  the  time  available  (10  milliseconds)  could  be  spent  just  copying  the 
information,  not  counting  the  overhead  associated  with  the  queue  and  space 
management.  If  we  add  to  this  figure  the  time  needed  to  process  the  packet 
(precedence  identification,  routing,  etc)  and  the  global  overhead  (polling  interrupts, 
computing  queue  statistics,  etc)  it  becomes  apparent  that  the  average  number  of 
pacKets/frame  together  with  the  copying  mechanism  used  are  the  major  independent 
parameters.  The  results  of  the  experiment  are  shown  in  figures  1.12  through  1.14.  In 
these  figure  the  Y-axis  indicates  the  average  packet  delay  for  each  class  (two  curves 
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ere  shown  in  each  figure)  and  the  X-axis  indicates  the  average  number  of 
packets/frame. 

Three  different  "copying"  schemes  were  tried.  Figure  1.12  shows  the  result  of 
using  the  standard  PDP-11  MOV  instruction  in  which  one  word  is  copied  between  two 
memory  locations.  The  figure  shows  that  although  high  priority  traffic  receives  a fairly 
constant  degree  of  service  over  a variable  traffic  volume  (represented  by  the  average 
number  of  packets/frame),  low  priority  traffic  tends  to  go  unattended  for  longer  and 
longer  periods  of  time,  represented  by  the  exponential  increase  in  the  average  delay. 
The  figure  shows  that  beyond  4 pacKets/frame  the  system  performance  deteriorates 
drastically. 

Figure  1.13  shows  the  result  of  using  a better  (hypothetical)  block  transfer 
instruction  that  might  be  implemented  on  (he  PDP-11  processors.  We  were 
conservative  in  our  approach  and  assumed  a speed-up  factor  of  4 (roughly  equivalent 
to  having  a 1.25  microsecond  MOV  instruction,  all  other  instructions  being  the  same). 
The  figure  shows  a slight  improvement  over  the  previous  result.  The  system  is 
capable  of  handling  an  average  of  5 packets/frame  before  the  low  priority  traffic 
delay  becomes  the  limiting  factor.  It  can  be  observed  that  the  delay  for  high  priority 
traffic  remains  constant  over  the  range  of  traffic  loads. 

Figure  1.14  shows  the  upper  bound  in  the  performance  of  the  PDP-1 1/20’s  used 
In  the  experiment.  Here  we  made  the  simplifying  assumption  that  specialized 
hardware,  running  concurrently  with  the  integrated  switch  processors,  could  take  care 
of  the  mundane  task  of  moving  blocks  of  data  between  memory  locations.  In  other 
words,  we  supressed  the  actual  copy  operation  (although  the  overhead  of  setting  up 
the  copy  loop  was  still  present  and  accounted  for).  The  figure  shows  a slight 
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I improvement  over  the  previous  results.  The  system  could  now  handle  an  average  of  6 

4 

t 

' pecKets/frame  before  the  average  delay  of  the  low  priority  traffic  became  impossible 

i 

I to  handle. 

I 

{ In  all  the  experiments  the  final  limit  was  reached  when  the  arrival  rate  so 

I 

outgrew  the  service  rate  that  critical  resources  liKe  queue  elements  and  packet  buffers 

I 

became  exhausted. 

i 

^ Skew  Impact 

One  of  our  early  asumptions  about  the  implementation  of  a SENET  was  that  no 
synchronization  between  input  and  output  lines  was  needed  or  advisable.  In  this 
experiment  we  attempted  to  measure  the  impact  of  the  skew  between  the  frame 
arrival  and  departure  times.  Figure  1.15  shows  the  effect  of  shifting  the  input  frame 
arrival  time  given  a fixed  frame  departure  time.  The  figure  shows  the  variation  in  the 
average  packet  delay  for  a load  of  4 packets/frame  (average).  The  packets  themselves 
were  drawn  from  the  same  distributions  as  in  the  previous  examples. 

Figure  1.15  indicates  that  varying  the  time  gap  between  the  frame  arrival  and 
departure  times  has  more  impact  on  the  high  priority  traffic.  The  shape  of  the  high 
priority  delay  curve  can  be  explained  as  follows:  For  small  values  of  the  skew  (1  or  2 
milliseconds)  very  few  packets  can  be  processed  in  time  to  be  transmitted  on  the  very 
next  output  frame.  Thus,  the  delay  grows  linearly  with  the  skew.  For  skew  values 


time,  reaching  a minimum  delay  with  a skew  of  6 milliseconds.  When  the  skew  exceeds 
6 milliseconds  the  packets  have  all  been  processed  and  they  sit  idle  in  the  output 
frame.  The  delay  therefore  grows  linearly  with  this  "excess”  skew  time. 
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The  figure  indicates  a variation  of  5.5  milliseconds  in  high  priority  traffic  delay 
or  (5.5/16.5)  33Z.  The  effect  on  the  low  priority  traffic  is  less  noticeable,  (2.5/20.5) 
127. 

Figures  1.16,  1.17,  and  1.18  present  the  results  of  similar  experiments  with 
decreasing  average  number  of  pacKets/frame:  3,  2,  and  1 packet/frame  respectively, 
j I It  can  be  observed  that  as  the  number  of  packets/frame  diminishes,  the  "optimal"  skew 

' value  decreases.  This  is  to  be  expected  since  the  load  has  decreased  and  most 

packets  will  be  processed  in  time  for  the  next  output  frame, 
i The  results  do  not  seem  to  justify  the  increased  complexity  of  a system  in  which 

■ the  skews  are  predetermined  as  would  be  the  case  in  a synchronous  network. 

Remember  that  this  is  just  the  transmission  delay  for  a single  node.  The  lifetime  of  a 

I packet  irtside  a node  is  always  greater  that  this  value  since  the  node  must  wait  for  an 

I 

[ acknowledgement  before  reclaiming  the  buffer  space,  in  which  case  the  impact  of  the 

! skew  could  very  well  be  negligible. 
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Real  Time  Traffic 

System  Parameters: 

Line  Speed:  1.544  Megabits/sec  (T1  carrier) 
Frame  Length:  10  Milliseconds 
Number  of  Channels:  2 (Full  duplex) 

Number  of  Processors:  3 (POP- 11/20) 

Frame  Departure  time:  0 milliseconds 
Channel  0 Frame  Arrival:  1 millisecond 
Channel  1 Frame  Arrival:  9 Millisecond 


As  in  the  Data  Transmission  experiments  two  different  types  of  memory-memory 
move  instructions  were  assumed.  In  the  first  type  each  individual  word  (16  bits)  of 
real  time  information  is  transmitted  individually  in  a software  loop.  The  second  type 
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assumes  a slightly  improved  instruction  set  in  which  a factor  of  4 speed-up  is  achieved 
by  doing  block  transfers.  In  the  experiment  we  considered  two  different  slot  sizes: 
small  (80  bit)  slots  which  could  be  suitable  for  voice  communications  and  large  slots 
(800  bits)  which  could  be  appropriate  for  facsimile  or  low  quality  image  transmission. 

The  following  table  summarizes  the  experimental  results.  The  numbers  indicate 
the  maximum  number  of  slots  that  could  be  transmitted  before  the  Real  Time 
Transmission  task  fell  behind  the  arrival  rate  and  had  to  give  up  working  on  a real  time 


packet. 

80bits/slot 

800  bits/slot 

1 word/move 

31 

15 

4 words/move 

34 

19* 

instruction 

a The  limit  here  was  dictated  by  the  frame  size.  20  slots  of  800  bits 

(16000  bits)  is  more  that  the  frame  size  (15440  bits). 

The  table  shows  that  transmitting  the  maximum  number  of  small  slots  using  a 
single  word/move  instruction  i.e.  31  slots,  only  uses  31*80-2480  bits  of  a frame  or 
2480/15440-16^  of  the  frame  capacity.  Transmitting  34  small  slots  using  a better 
move  instruction  only  utilizes  34*80-2720  bits  or  2720/15440-17.5^  of  the  frame;  a 
rather  small  improvement. 

The  transmission  of  large  slots  presents  a different  picture.  Even  when  a single 
word/move  instruction  is  used,  the  system  can  transmit  15*800  bits/frame  or 
12000/15440-78Z  of  the  frame  capacity.  Using  the  faster  block  transfer  instruction 
indicates  a 100^  utilization. 

The  dramatic  difference  between  the  results  indicates  that  the  limiting  parameter 
is  the  number  of  slots  rather  that  the  slot  size.  In  other  words,  the  overhead  in 
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setting  up  the  loop  for  transmitting  a slot  is  the  factor  that  dominates  the  switch 
capacity  for  real  time  transmission. 

The  following  analysis  determines  this  overhead  component.  The  load  on  a 
processor,  in  the  absence  of  other  traffic  types,  can  be  modeled  by:  C’  N * (C  X) 
where  C’  is  the  overhead  incurred  by  the  processors  in  doing  non-real  time  related 
tasks  (inspecting  queues,  polling  interrupts,  collecting  statistics,  etc.)  C*X  describes  the 
real  time  transmission  loop.  C is  a constant  overhead  spent  in  loop  control  and  X is  the 
actual  slot  movement  time  (X  is  variable,  depending  on  the  type  of  move  instruction 
used).  N is  the  number  of  slots/frame. 

The  following  equation  describes  the  improvement  in  the  switch  capacity  as  a 
result  of  using  the  block  transfer  instruction  on  small  slots: 
C’  +31  * (C  + X)  - C’  + 3^  * (C  + X / 4)  from  which  we  obtain,  (after  some  algebraic 
simplifications):  C - 7.5  * X i.e.  the  overhead  of  moving  a real  time  slot  is  7.5  times  the 
cost  of  the  actual  movement.  If  we  consider  that  a POP- 11/20  takes  5 microseconds  to 
move  a word  (16  bits)  from  memory  to  memory  then  the  time  to  move  a small  slot  is: 

X ■ SObits  * lword/16bits  * 5microseconds/word  ■ 25microseconds. 

The  loop  overhead  time,  C,  is  therefore  25  * 7.5  - 187.5  microseconds  and  the 
total  time  to  move  a slot  is  C + X - 187.5  + 25  - 212.5  microseconds.  The  time  needed 
to  transmit  31  slots  is  31  * 212.5  - 6587.5  microseconds. 

The  latter  figure  indicates  that  it  takes  66t  of  the  time  to  transmit  16^  of  the 
frame  (17.57  using  the  block  move  instruction). 

The  analysis  of  the  improvement  in  the  switch  capacity  when  using  large  slots  is 
described  by;  C*  + 31  ♦ (C  ♦ X)  ■ C’  +15  * (C  + 10  ♦ X)  (C+lOX  is  the  time  required  to 
move  slots  that  are  10  times  longer). 
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The  equation  yields:  C > 7.45  * X which  is  consistent  with  the  previous  result. 
The  total  time  to  move  a large  slot  is:  C + 10  ♦ X ■ 17.45  ♦ X - 17.45*25  ■ 436 
microseconds. 

The  transmission  of  15  large  slots  takes:  436  * 15  - 6550  microseconds.  i.e.  it 
takes  66^  of  the  time  to  transmit  78^  of  the  frame  capacity. 

The  evidence  above  shows  that  the  system  is  better  utilized  during  the 
transmission  of  large  size  real  time  slots,  and  that  any  significant  improvements  will 
have  to  come  from  a reduced  overhead.  In  other  words,  software  table  look-ups  and 
loops  can  not  perform  a satisfactory  job.  Later  in  this  Section  we  will  describe  some 
ideas  that  should  be  considered  in  the  implementation  of  firmware  or  hardware  real 
time  transmission  schemes. 

Mixed  Real  Time/Data  Traffic 

Figure  19  shows  the  effect  of  various  levels  of  real  time  traffic  on  the  average 
packet  delay.  Notice  that  for  this  particular  experiment  we  were  able  to  utilize  4 
processors  (PDP-1 1/20's)  instead  of  the  usual  3 processors  as  indicated  in  the 
previous  experiment  discussions.  Moreover,  the  presence  of  the  extra  processors 
allowed  us  to  process  the  real  time  traffic  in  a slighiy  different  manner.  For  this 
experiments  we  divided  the  real  time  copy  task  into  two  subtasks/frame  i.e.  two 
processors  could  now  co-operate  in  the  disection  of  the  real  time  packets  for  each 
input  frame. 

The  figure  shows  that  the  real  time  load  impacted  the  low  priority  packets  more 
seriously  than  the  high  priority  packets.  Increasing  the  number  of  real  time  slots 
resulted  in  an  increase  in  the  average  packet  delay,  as  was  to  be  expected. 

The  important  result  that  can  be  observed  from  this  experiment  is  the  ratio 
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between  an  increase  in  the  number  of  real  time  slots  and  the  reduction  in  the  average 
number  of  pacKets/frame  that  could  be  processed.  If  we  observe  the  solid  tines, 
representing  the  high  priority  traffic  delays,  we  can  estimate  a ratio  of  7.5  real  time 
slots/packet  (going  from  10  to  40  real  time  slots  resulted  in  a reduction  in  the  number 
of  packets/frame  from  7 down  to  3,  i.e.  (40-10)/(7-3)-7.5  slots/packet).  If  we  ignore 
the  slight  variation  in  packet  delay,  the  dotted  lines  (low  priority  traffic)  associated 
with  32  and  40  real  time  slots  indicate  a decrease  of  1 packet/frame  (from  4 down  to 
3)  that  is  to  say,  a ratio  of  8 slots/packet. 


1.3.3.  Alternative  Architectures;  C.mmp  Machine  vs.  Hydra  Machine 


The  Hydra  operating  system  currently  in  use  on  C.mmp  creates  a very  different 

; 

! 

i machine  from  the  user’s  point  of  view  than  what  is  available  on  the  bare  hardware, 

i Very  early  in  the  design  stage,  we  had  to  decide  whether  to  work  under  Hydra  (i.e.,  on 

i 

the  Hydra  machine),  or  on  the  bare  C.mmp  hardware.  The  reasons  for  the  choice  made 
are  given  here,  after  some  review  of  the  basic  features  of  the  two  machines. 

i I 

I ; Design  Features  of  the  C.mmp  Machine 

1 

The  most  important  feature  of  C.mmp,  of  course,  is  the  set  of  16  independent, 
asynchronous  processors,  sharing  a common  main  memory.  In  addition,  there  are  a 
! total  of  16  different  memory  ports,  allowing  up  to  1024K  words  each.  The  processors 

I 

I and  ports  are  joined  by  a 16  x 16  crosspoint  switch,  allowing  a maximum  of  16 

I 

1 

■ simultaneous  processor-memory  conversations.  In  addition  to  the  shared  memory,  each 

i 

processor  has  4K  words  of  local  memory  accessible. 

I 

I Because  of  the  limitations  of  the  16  bit  address,  not  all  of  memory  is  directly 

addressable  by  a processor  at  any  one  time.  Memory  is  organized  into  pages  of  4K 
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words  each  (8K  bytes).  A set  of  relocation  registers  connected  to  each  processor 
allows  a maximum  of  8 pages  to  be  addressed  at  once,  for  a total  of  64K  words.  This 
small  address  space  has  an  enormous  effect  on  the  construction  of  large  programs  on 
C.mmp,  since  they  must  be  organized  as  segments  of  4K  words  each.  Also  the  user 
must  supply  his  own  convention  for  segmentation,  overlays,  and  the  like. 

Each  processor  has  the  ability  to  start,  stop,  or  interrupt  any  or  all  of  the  other 
processors  by  means  of  interprocessor  interrupts  (IPIs).  These  IPIs  can  occur  at 
different  priority  levels,  and  allow  fast  interprocessor  communication.  There  are  also 
four  levels  of  operation  priveleges:  the  00-space,  01-,  10-,  and  11 -spaces.  The  00- 
space  has  the  lowest  priveleges  (i.e.,  user-space),  while  the  1 1 -space  has  the  highest 
(e.g.,  operating  system).  Each  space  has  its  own  set  of  8 relocation  registers  which 
are  independent  of  the  relocation  registers  in  any  of  the  other  spaces. 

The  I/O  devices  arc  attached  to  thee  Unibusses  of  specific  processors.  This 
means  that  any  I/O  interrupts  can  only  come  to  the  processor  which  owns  the  device 
(herKe  the  need  for  fast  interprocessor  communications).  There  is,  however,  one 
"device"  common  to  all  the  processors:  a 60-bit  global  clock,  which  has  a resolution  of 
4 microseconds.  In  addition,  each  processor  has  an  interval  timer  clock  which  is  driven 
off  this  global  clock,  with  resolution  of  16  microseconds. 

Finally,  C.mmp  allows  the  use  of  microcoded  POP-1  l/40Es  on  the  system. 
Eventually,  a maximum  of  twelve  40Es  will  be  included  in  the  system,  with  the  other 
four  processors  being  PDP-ll/20s.  Thus,  it  will  be  possible  to  implement  special- 
purpose  instructions  for  particular  tasks  by  use  of  the  writable  microstore. 

Current  Status  at  C-mmo  Machine 

Currently  there  a total  of  nine  processors  connected  to  C.mmp,  five  POP- 11 /20s 
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I 

I end  four  PDP-ll/40Es.  However,  the  integration  of  the  40E  processors  into  C.mmp  is 

I 

not  yet  complete.  Some  problems  have  been  discovered  in  the  original  interface  design 
I which  usually  keep  them  from  performing  reliably  on  C.mmp.  Hence,  the  model  40s  are 

I 

I usually  partitioned  out  of  the  running  C.mmp.  The  total  number  of  running  processors 

1 

is  usually  around  five  (all  the  11 /20s),  though  the  number  has  occasionally  been  as  low 

as  three,  and  as  high  as  seven  or  eight. 

i 

Currently,  there  are  a total  of  680K  words  of  memory  on  16  ports.  Occasional 

I 

parity  errors  or  bad  ports  are  detected,  but  the  memory,  by  and  large,  is  reliable.  The 
rest  of  the  hardware  of  C.mmp  (IPls,  global  60-bit  clock,  interval  timers,  00-11  spaces, 
relocation  registers,  and  switch  concurrency)  are  also  operating  relatively  reliably. 
Design  Features  of  Hydra  Machine 

Hydra  is  the  operating  system  designed  for  C.mmp.  Its  philosophy  is  to  create  a 
general-purpose  machine  out  of  C.mmp  in  a timesharing  environment.  Hydra  hides 
many  of  the  details  of  the  underlying  machine  that  are  unnecessary  to  know  about 
from  a user’s  point  of  view.  The  operating  system  also  provides  the  tools  for  creating 
a highly  protected  system  based  on  the  use  of  capability  lists,  access  rights  on 
capabilities,  etc. 

Hydra  is  divided  into  two  parts:  the  subsystems  and  the  Kernel.  The  Hydra 
Kernal  is  a set  of  routines  which  cover  up  all  the  "dangerous”  aspects  of  the  machine. 
These  routines  are  not  the  operating  system,  but  rather  the  tools  with  which  to  build 
the  operating  system.  The  Kernal  operates  in  the  privileged  11 -space;  users  operate 
in  the  unprivileged  00-space.  Certain  "dangerous*  instructions,  and  all  interrupts,  are 
handled  only  in  the  Kernal. 

' Outside  the  Kernal  are  components  called  "subsystems",  which  implement  all  the 
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necessary  functions  of  a timesharing  operating  system:  a file  system,  command 
language,  scheduler,  compilers,  utilities,  resource  allocater,  etc.  These  subsystems  do 
not  have  access  to  the  underlying  machine  ~ only  the  Kernel  has  that  capability.  In  a 
sense,  they  are  all  user  programs,  and  in  principle,  several  disjoint  sets  of  subsystems 
could  be  operating  concurrently.  Hence,  "Hydra”  is  a generic  name  applied  to  the 
Kernal  and  some  set  of  subsystems  (however,  there  exists  only  one  set  currently). 

Many  more  "processes"  are  allowed  to  exist  than  processors.  The  Policy  Module 
subsystem  (by  using  tools  in  the  Kernal)  allocate  processors  to  processes  via  a specific 
scheduling  policy.  The  user  cannot  guarantee  when  a process  will  get  scheduled,  or 
how  long  it  retains  control  of  the  processor  --  this  is  a result  of  Hydra  being  a 
timesharing  machine. 

Pages  for  a user  process  are  allocated  by  the  Kernal  — the  user  has  no  control 
over  which  physical  pages  are  allocated  to  him,  or  more  importantly,  no  control  over 
which  pages  go  into  which  ports.  This  means  that  he  has  no  control  over  the  amount 
of  memory  contention  that  takes  place  when  several  processors  are  all  making 
continuous  memory  requests.  They  will  all  run  significantly  faster  when  the  memory 
requests  are  to  different  ports,  thus  utilizing  the  concurrency  of  the  crosspoint  switch. 
A high  memory  contention  rate  causes  degradation  of  processor  speed. 

DCA  Communications  Task  on  C.mmo  Machine 

There  are  a number  of  strong  advantages  for  using  the  C.mmp  machine  for  the 
communications  task.  Basically,  it  is  the  most  appropriate  machine  for  running  a real- 
time program  on.  One  has  complete  access  to  the  global  clock,  interval  timers,  and 
IPIs. . Precise  timing  is  possible  because  interrupts  are  fielded  directly  --  hence  polling 
it  unnecessary.  In  addition,  pages  can  be  allocated  specifically  to  minimize  memory 
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contention  and  thus  increase  processor  performance.  User  microcode  can  be  easily 
irtserted  for  special-purpose  instructions.  In  short,  this  is  just  the  advantage  of  being 
able  to  get  to  the  all  the  hardware  for  maximum  performance  and  precision. 

However,  there  are  some  major  practical  disadvantages  in  using  the  bare  C.fflmp 
machine.  Specifically,  there  is  virtually  no  support  available  for  running  on  a stand- 
alone C.mmp.  This  means  that  the  user  would  have  to  write  his  own  software  for 
handling  gH  interrupts  (I/O,  IPIs,  hardware  errors,  etc.),  drivers  and  error-handlers  for 
all  desired  I/O  devices,  recovery  procedures  from  exceptional  conditions,  and  so  on.  In 
esserKe,  this  means  writing  a mini-operating  system. 

In  addition  to  writing  a great  deal  of  low-level  code,  one  must  also  worry  about 
debugging  such  notoriously  difficult -to-debug  code  as  interrupt  handlers.  Complicating 
the  situation  further  is  the  relatively  small  amount  of  time  each  day  when  one  can  sign 
up  for  exclusive  use  of  C.mmp  — a small  point,  but  one  which  could  have  a great  effect 
on  the  length  of  time  before  a body  of  software  is  up  and  running. 

DCA  Communications  Task  on  the  Hydra  Machine 

The  main  advantages  of  running  under  Hydra  are  exactly  those  features  without 
which  it  would  be  very  difficult  to  run:  a file  system,  process  starting/stopping, 
availability  of  I/O  support,  exceptional  condition  handling,  interrupt  handlers, 
debugging  aids.  There  is  also  a support  group  for  Hydra-related  functions  which  are 
available  for  fixes  or  improvements  in  the  operating  system.  Finally,  there  is  the 
advantage  that  most  of  the  debugging  can  be  carried  out  in  a timesharing  mode  with 
other  users,  hence  the  availability  of  the  machine  is  greatly  increased. 

However,  there  are  some  major  disadvantages  in  using  the  Hydra  machine  for 
the  communications  task.  All  interrupts  are  intercepted  by  the  Kernel  — the  user  is 
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never  allowed  to  field  an  interrupt  directly  (which  is  a distinct  disadvantage  for  a real- 
time program).  The  best  possible  compromise  is  a form  of  polling,  where  the  Kernel 
interrupt-handler  sets  particular  bits  for  specific  interrupts. 

Hydra  allows  lots  of  high-level  abstractions:  processes,  semaphores,  ports  and 
messages,  capabilities,  etc.  However,  the  use  of  these  abstractions  is  very  costly  in 
processor  time  (e.g.,  a context  swap,  where  a processor  is  switched  from  one  process 
to  another,  takes  about  20  ms.  on  a PDP-11/20).  Hence,  the  use  of  these  natural,  but 
expensive,  abstractions  is  prohibited  by  the  real-time  nature  of  the  program. 

Another  problem  is  that  Hydra  insists  on  doing  things  underneath  any  user 
program.  For  example,  time  slice  timeouts,  I/O  interrupts,  and  periodic  scheduling 
interrupts  (to  allow  a processor  to  swap  to  a higher  priority  process)  are  all 
performed  invisibly  to  a user  program.  However,  they  all  take  processor  time  in 
potentially  deadline  situations  (such  as  getting  Class  I traffic  into  an  output  frame  in 
time).  Several  things  were  done  to  reduce  or  eliminate  these  problems:  (1)  Runs  were 
done  on  a dedicated  system  with  infinite  time  slices  — no  other  users  were  allowed  to 
run  at  the  same  time;  (2)  specific  sections  of  the  operating  system  were  patched  to 
eliminate  periodic  low-level  activity;  and  (3)  the  system  was  designed  so  that  no  I/O 
activity  took  place  during  the  actual  running  of  the  simulation  (though  it  was  allowed 
during  input  and  analysis). 


For  overwhelmingly  practical  reasons,  the  Hydra  machine  was  selected  to  be 
used.  Looking  back,  it  is  doubtful  that  we  would  even  be  close  to  having  a running 
system  now  if  the  C.mmp  machine  were  chosen.  Much  low-level  code  concerned  with 
the  smallest  details  of  the  POP-1  Is  would  have  to  have  been  written  and  debugged  — 
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end  debugging  of  time-dependent  interrupt  routines  can  be  tedious  and  time- 
consuming,  even  for  an  expert. 

Yet  the  choice  of  the  Hydra  machine,  white  pragmatic,  had  many  bad  effects.  We 
tried  to  make  use  of  Hydra  in  a way  never  intended  by  its  designers  (dedicated 
machine,  patching  out  annoying  interrupts,  allowing  infinite  time-slices,  etc.).  Moreover, 
the  choice  dictated  a fundamental  part  of  the  design:  interrupts  had  to  be  polled  rather 
than  handled  directly.  Polling  has  other  advantages,  such  as  not  having  to  save  state, 
but  we  were  barred  from  even  considering  precise  handling  of  interrupts.  Hydra  also 
has  a strong  effect  on  the  time  required  to  load  a relocation-register:  the  basic 
machine  instructions  require  10-20  microseconds,  while  the  corresponding  Hydra 
furKtion  requires  300  microseconds.  We  have  been  able  to  largely  avoid  the  address 
space-problem  by  structuring  the  page-configurations  so  that  dynamic  relocation- 
register  loading  is  not  necessary.  However,  Keeping  this  structure  may  not  be  possible 
if  the  system  size  increases  substantially.  In  this  case,  dynamic  relocation-register 
loading  would  probably  be  the  required  solution  — one  that  would  be  much  more 
expensive  under  Hydra  than  on  the  bare  C.mmp  machine. 

* a 

1.3.4.  Alternative  Design  for  Class  1 Traffic  Transmission 

Class  I traffic  (real-time)  is  essentially  a "scatter-write"  operation.  That  is.  Class 
I slots  arrive  contiguously  in  an  incomimg  frame  at  a node,  but  are  transmitted  out  on 
other  channels  in  slot  positions  independent  of  their  incomimg  frame  slot  position. 
(Alternatively,  the  operation  can  be  viewed  as  a "gather-read",  where  the  contiguous 
slots  in  each  outgoing  frame  are  gathered  from  various  input  frames  and  slot 
positions.)  The  Class  I slots  may  of  course  be  of  varying  sizes,  but  they  are  assumed 
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to  contain  an  integral  number  of  characters  and  to  remain  fixed  in  size  for  the  duration 
of  a connection. 

The  current  implementation  for  the  "scatter-write"  operation  is  by  software.  An 
input  frame  is  received  and  placed  in  contiguous  memory  by  the  input  Line  Interface 
Device  (LID).  Slots  are  then  copied  by  software  to  appropriate  buffer  positions  to 
create  contiguous  output  slots.  (Note  that  copying  was  performed,  rather  than  simply 
passing  a pointer.  The  reason  for  this  is  that  the  output  LID  was  already  assumed  to 

. be  following  a two-level  structure  of  pointers.  Passing  slot  pointers  would  necessitate 

! assuming  a hardware  LID  design  involving  three  levels  of  pointer  nesting:  a pointer  to 

i 

! the  Channel  Command  Vector  (CCV);  pointers  in  the  CCV  to  packets;  pointers  in  the 

Class  I packet  to  individual  slots.  This  seemed  like  too  much  to  assume  for  a hardware 

I 

i 

j device.) 

i 

I The  results  of  simulation  runs  for  Class  I traffic  indicate  that  Class  I transmission 

) by  software  is  an  expensive  proposition.  Moreover,  block  transfers  or  even  passing 

pointers  would  not  gain  very  much  because  of  the  relatively  large  amount  of  overhead 

i 

, per  slot,  especially  small  slots,  spent  in  table-lookups,  queue  operations,  etc.  Since  the 

i 

Class  I transmission  is  basically  a straightforward  operation,  one  would  like  to  have  the 
software  program  of  a node  simply  simulate  the  role  of  a telephone  operator,  setting 

I 

I up  and  breaking  down  connections  as  needed,  but  not  involved  with  the  actual  transfer 

I 

I of  information  along  the  connections. 

I 

! One  approach  which  satisfies  this  condition  would  use  a special-purpose 

I microprocessor  connected  directly  to  the  input  line  and  able  to  access  the  output 

( 

I buffers  in  memory.  This  microprocessor  would  refer  to  a table  to  obtain  the 

destination  buffer  address  and  slot  length  for  each  incomimg  Class  I slot.  Thus,  as  the 
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Class  I packet  begins  to  arrive,  the  microprocessor  would  read  the  first  entry  in  the 
table  to  determine  starting  address  and  length,  and  then  store  the  corresponding 
number  of  incomimg  bytes  into  sequential  locations  starting  with  the  stated  address. 
Then  the  table  would  be  referenced  again  for  the  destination  and  length  of  the  second 
slot,  and  so  on.  Thus,  instead  of  depositing  incomimg  Class  I slots  into  contiguous 
locations  in  memory  (i.e.,  an  "input  buffer",  which  is  the  assumption  in  the  current 
simulation  program),  this  microprocessor  would  be  constantly  referencing  the  table  in 
memory  and  storing  the  incomimg  slots  in  their  appropriate  positions  in  the  output 
buffers.  The  net  effect  of  such  an  operation,  with  a microprocessor  for  each  incomimg 
channel,  is  the  creation  of  contiguous  output  Class  I buffers,  ready  for  sending  out  on 
the  next  link. 

The  primary  task  for  the  node  software  here  is  updating  the  table  as  changes 
occur  in  the  Class  I packet  composition  (e.g.,  normal  setting  up  and  breaking  down  of 
connections).  The  situation  is  complicated  slightly  by  the  need  for  a ring  of  three 
buffers  for  Class  I traffic  --  to  ensure  that  an  incomimg  frame  is  not  switched  to  a 
different  buffer  in  mid-stream  (see  Appendix  I)  — but  the  added  complexity  is 
relatively  minor. 

The  operation  of  several  microprocessors  (one  for  each  input  channel)  accessing 
main  memory  continuously  could  become  a major  bottleneck.  For  example,  assume  that 
each  microprocessor  obtains  a 16-bjt  word  from  the  LIO,  then  about  10.4  microseconds 

elapse  between  words  (at  the  rate  of  a T1  carrier,  15440  Kbps).  [)uring  each  10.4 

\ 

mIcrosec.  interval,  the  microprocessor  will,  in  the  worst  case,  have  to  perform  a read 
access  on  the  slot-destination  table,  and  a memory-write  of  the  incomimg  word  to  the 
specified  destination.  Of  course,  most  slots  would  be  more  than  1 word  long,  so  the 
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table-read  would  only  have  to  be  done  intermittently.  But  a worst-case  analysis  would 
have  each  of  the  bnn  microprocessors  performing  a memory  read  and  write  cycle 
every  10.4  microsec.  In  addition,  the  output  LID  for  each  channel  would  be  reading  a 
buffer  at  the  rate  of  one  read  access  every  10.4  microsec.  Hence,  we  see  that 
n ( 2R  ♦ W ) <.  10.4 

where 

n is  the  number  of  channels 
R is  the  memory  read-access  time 
W is  the  write  cycle  time. 

For  example,  if  R-0.5  microsec.  and  W-1.0  microsec.,  then  in  the  worst  case  no 
more  than  five  T1  lines  could  be  handled  by  the  memory.  Note  that  this  also  assumes 
high . priority  for  the  microprocessor  memory  accesses,  implying  that  any  additional 
source  of  memory  accesses  (e.g.,  instruction  fetching)  would  be  degraded.  Interleaving 
memory  or  providing  multiple  ports  could  alleviate  this  possible  bottleneck. 


^ 

J 
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lA  Appendix  1 ; Class  1 Traffic.  Further  Considerations 

, There  are  some  further  issues  and  problems  that  must  be  dealt  with  before  the 

( 

Slotted  Envelope  design  can  be  completely  defined.  This  section  describes  some 

i 

special  problems  associated  with  the  handling  of  Class  I traffic  and  suggests  some 

! 

I possible  approaches  to  resolving  them. 

i 

1.4.1.  Class  I Reservation  Protocol 

I The  maintenance  of  routing  tables  is  one  of  the  outer  loop  functions  that  will  be 

' studied  during  the  second  year  of  the  contract,  in  this  section  we  will  present  some 

preliminary  thoughts  on  the  matter. 

! Reservation  of  a Class  I channel  across  the  network  would  be  done  in  several 

! 

' stages.  First,  upon  call-initiation,  reservation-request  messages  would  be  sent  to 

^ determine  a feasible  path  through  the  network  for  the  connection.  Nodes  in  the  path 

I 
t 

woul'  * ! ♦serve  a slot  in  the  Class  I portion  of  their  frame  for  the  call,  although  the  slot 

I 

I would  I'nt  actually  be  allocated  it  until  the  call-initiation  was  successfully  completed 

I 

1 • 

I (i.e.,  the  called  party  answered  the  phone).  At  this  point,  the  reserved  slots  would  be 

I 

allocated,  on  a pairwise  basis  on  the  nodes  along  the  path  of  the  call.  The  Class  I 
I reservation  tables  used  by  the  nodes  are  depicted  in  Figure  A.1. 

i'  Since  nodes  at  both  source  and  destination  ends  of  a specific  link  would  have  to 

agree  in  advance  on  changes  to  the  Class  I portion  of  frames  traversing  that  link,  some 
sort  of  slot  allocation/deallocation  protocol  would  be  required  which  would  enable 
agreement  on  when  a Class  I change  is  implemented  as  well  as  what  the  change  is. 

' The  addition  of  a time  requirement  complicates  the  situation  since  delays  or  line  errors 
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could  occur,  forcing  timeout  and  retransmission  of  the  CCIS  packets  in  the  presence  of 
a potential  deadline. 

Two  basic  strategies  in  the  protocol  are  possible.  The  first,  which  could  be 
termed  the  time  relative  approach,  is  to  have  the  CCIS  always  reference  a change-time 
(frame  number)  that  is  a constant  time  in  the  future  from  the  frame  containing  the 
"request  for  allocation”  (ALLOC)  packet.  The  responding  node  would  acknowledge  the 
ALL(X!  packet  with  an  acknowledgement  (ACK)  packet  containing  the  same  information. 
Retransmission  of  the  ALLOC  packet  (with  a new  proposed  change-time)  would  occur  if 
the  ACK  information  did  not  match,  or  no  ACK  arrived  before  timeout.  The  advantage 
of  this  approach  is  that  a deadline  situation  is  avoided,  since  any  retransmission  would 
put  off  the  change-time  by  an  amount  equal  to  the  delay  encountered  The  danger 
with  this  approach  is  that  an  indefinite  postponement  of  the  change  could  occur,  as 
illustrated  in  Figure  A.2(a).  Node  A originates  the  ALLOC  packet  in  frame  1,  with  a 
change-time  of  +7  relative  to  1,  or  8.  The  ACK  packet  from  node  8 is  delayed,  so  A 
times  out  and  resends  the  ALLOC  packet  in  frame  6 with  a new  change-time  of  13 
(6+7).  This  ALLOC  message  crosses  with  the  delayed  ACK  from  B.  Node  A must  now 
send  anothe"  ALLOC  packet  since  the  ACK  information  does  not  match,  while  B must 
also  respond  to  the  new  ALLOC  packet.  An  ”out-of-phase"  situation  like  this  could 
continue  indefinitely  and  cause  an  unpredictable  delay  in  establishing  a Class  I change 
between  two  nodes. 

The  second  approach,  termed  time  absolute,  would  have  retransmissions 
reference  the  same  future  time  as  originally  proposed  in  the  first  ALLOC/DEALLCX) 
CCIS  packet.  No  ”out -of -phase”  situation  is  now  possible,  but  the  new  danger  is  that 
of  or>ly  partially  completing  the  handshaking  procedure  before  a deadline  expires.  This 
it  illustrated  in  Figure  A.2(b). 
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In  both  these  strategies,  the  chance  of  error  can  be  reduced  by  appropriate 
selection  of  timeout  durations,  change-time  increments,  and  CCIS  pacKet  priorities. 
Improvements  in  these  strategies  are  obviously  also  possible,  though  it  is  difficult  to 
ensure  that  any  protocol  is  absolutely  foolproof  against  all  possible  situations. 


1.4.2.  Oass  1 Slot  Slippage  and  Overwrite 

As  mentioned  before,  in  Section  2.2,  there  are  some  problems  that  might  arise 
during  the  transmision  of  Class  I traffic.  These  problems  are  best  illustrated  by  an 
example.  The  following  two  paragraphs  first  explain  the  figure  and  then  illustrate  the 
slippage/overwrite  effect. 

Figure  A.3  schematically  shows  the  flow  of  information  through  a single  input 
channel  (IN)  and  a single  output  channel  (OUT).  Successive  input  frames  are  labelled 
(INI,  IN2,  etc.),  as  are  the  output  frames.  For  clarity,  the  Class  II  portion  of  each 
frame  has  been  omitted,  and  only  four  Class  I slots  (a»  ^ c,  and  d)  are  shown  in  a 
frame.  To  simplify  the  explanation,  it  is  assumed  that  all  four  incoming  Class  I channels 
all  go  out  on  the  same  output  channel.  In  general,  frames  on  input  and  output  channels 
will  not  be  synchronized.  In  this  example,  the  output  frame  lags  the  input  frame  by  a 
duration  of  fl  (-t2-tl).  Finally,  the  two  buffers  of  the  OUT  channel  are  explicitly 
shown:  during  frame  OUTl,  for  example,  buffer  1 is  being  filled,  while  buffer  2 is  being 
transmitted  and  is  locked  out  (indicated  by  the  heavy  dashed  line). 

At  time  tl  in  Figure  A.3,  incoming  frame  INI  has  been  completely  read  in  and  is 
made  available  for  processing.  The  switch  program  then  has  until  t2  to  transfer  slots 
into  buffer  1 of  OUT;  at  t2,  the  buffers  are  switched  and  any  subsequent  transferring 
must  go  into  buffer  2.  In  the  example,  assume  that  the  switch  processor’s  speed  and 
the  lag  duration  fi  are  such  that  the  switch  program  can: 
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(1)  always  transfer  slots  a and  ^ 

(2)  usually  transfer  slot  c,  and 

• never  transfer  slot  ^ 

~ 4 V.  during  the  interval  fi  before  the  buffers  are  switched  (at  t2,  t4,  etc.).  If  the  frame 

duration  is  denoted  by  F,  and  the  frame  lag  by  fi,  then  the  delay  through  the  node  for 
Class  I channels  a and  b is  F-I-/1.  The  delay  for  channel  ^ which  is  always  delayed  an 
extra  frame,  is  2F*fi.  Channel  c,  however,  can  experience  either  of  these  delays, 
I depending  on  whether  each  slot  c^  gets  transferred  into  the  earlier  buffer  or  later 

.1  buffer.  At  the  transitions  between  the  two  delays  for  channel,  c,  there  occurs  either  a 

• . 

j slip  (if  the  delay  is  increasing),  or  an  overwrite  (if  the  delay  is  decreasing).  Figure  A.3 

' shows  a slip  in  channel  c during  the  0UT2  frame.  Additionally,  an  overwrite  would 

I have  occurred  in  OUT3,  if  cq  had  been  transferred  during  the  {t5,t6)  interval, 

i While  occasional  slips  or  overwrites  might  not  have  any  strongly  noticeable 

j effect  on  voice  conversations,  other  real-time  connections,  such  as  encrypted  voice, 

' can  be  seriously  affected  if  an  error  occurs  when  an  encoding  key  is  being 

transmitted. 

It  should  be  noted  that  the  specific  effect  described  occurs  because  of 
variations  in  response  time  to  a frame  arrival.  This  is  certainly  more  likely  in  a 
software  implementation  of  "scatter-write"  (e.g.,  a frame-arrival  interrupt  may  not  be 
taken  immediately  because  of  another  higher  priority  interrupt);  a hardware  realization 
would  be  much  less  susceptible  to  these  variations,  though  it  would  still  depend  on  the 
source  node  to  time  the  frames  precisely. 

Even  if  all  timing  and  response  times  are  precise,  an  overwrite  could  still  occur 
in  a perfectly  innocent  way:  If  slots  b and  c were  deleted  after  input  frame  INI  (due  to 
rrormal  disconnection),  then  frame  IN2  would  only  have  two  active  slots  to  transfer,  £ 
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I 

and  i which  could  be  done  in  the  interval  (t3,t4).  Hence,  d’s  delay  would  decrease 
from  2F+/?  to  F*/S,  causing  an  overwrite  in  frame  0UT2. 

A solution  to  these  problems  is  simply  to  use  an  extra  buffer,  so  that  an  input 
frame  is  never  switched  to  a new  buffer  in  mid-stream.  The  penalty  for  this  is  that 
the  cross-node  delay  for  each  Class  1 channel  is  now  2F+^  units.  (A  hardware 
implementation,  by  doing  the  transfer  as  the  data  is  arriving,  could  reduce  this  to  F+/?.) 

Furthermore,  buffer  selection  would  be  decided  by  the  arrival  time  of  an  input  frame;  j 

i 

, if  the  input  channel  and  output  channel  were  almost  synchronized,  any  relative  drift 

between  the  clocks  at  the  two  nodes  could  cause  a discontinuity  in  buffer  selection 

I 

! (compared  to  the  normal  cyclic  order),  and  hence  a slip  or  an  overwrite  for  all  the 

' Class  I channels  on  that  input  frame.  This  is  expected  to  occur  rarely,  however, 

k 

I . especially  if  the  relative  drifts  between  nodal  clocks  is  small. 

I 

t 
I 


i 


I 

I 

i 

I 

I 

I 

I 

i 

I 


I 
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Ttw  following  (Figures  A.4,  A.5,  A.6)  is  an  example  of  the  dialogue  with  the 
Script  Driver  for  specifying  the  node  and  traffic  characteristics. 

- For  those  parameters  having  relatively  constant  values,  defaults  are 

built  in.  These  default  values  appear  enclosed  in  angular  brackets 
after  the  question. 

- Run  Number  is  a number  used  for  identifying  the  experiment  and 

associating  the  statistics  printout  with  the  parameters  specified  by 
the  user. 

- The  run  durations  to  obtain  steady  state  and  after  steady  state  are 

specified  in  number  of  frames  sent  on  any  single  line. 

- If  two  or  more  channels  are  identical,  only  one  need  be  specified.  The 

Script  Driver  will  duplicate  the  channel  specifications  for  additional 
instances  of  a previously  specified  channel. 

- The  distributions  for  packet  lyP^s  are  given  as  percentages  and  their 

interpretation  is  efxpMM^  tn  SMion  3.1. 

- The  average  number  of  packets  per  channel  is  A*T  where  T is  the  frame 

duration  and  A is  the  mean  arrival  rate  of  a Poisson  process. 

. ' - Packet  lengths  are  specified  in  number  of  bytes.  The  length  is  drawn 

from  a uniform  distribution  limited  by  the  maximum  and  minimun 
values  specified  by  the  user. 

- The  distribution  of  packet  destinations  indicates  the  average  fraction  of 

packets  that  go  to  each  output  channel. 

- The  real  time  packet  composition  is  indicated  explicitly  so  that  a desired 

pattern  can  be  specified. 

- The  frame  departure  and  arrival  times  are  relative  times.  Thus  one  of 

the  event  occurs  at  time*K)  and  other  events  occur  at  the  indicated 
relative  times. 

( 

i The  entries  are  displayed  in  a coded  format.  This  is  to  enable  the  user  to 

identify  a parameter  by  tts  coda  manta. 

l' 

• The  codes  are: 

I 

I 

R Run  number 
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S Seeds  for  rerKtom  number  generator 
B number  of  frames  Before  the  steady  state  is  obtained 
H How  long  the  system  is  to  run 
Nx  specification  for  channel  x 

Tx  speed  of  channel  x 

Lx  maximum  amd  minimum  packet  length  (in  bytes)  on  channel  x 
Px  packet  type  percentages  on  channel  x 
Dx  packet  Destinations  percentages  from  channel  x to  0,1,2  3 

V Composition  of  the  real  time  packets. 

I  events  vector  i.e  relative  times  for  varios  events 
event  definitions: 

0 frame  arrival  on  Channel  0 

1 frame  arrival  on  Channel  1 

2 frame  arrival  on  Channel  2 

3 frame  arrival  on  Channel  3 

4 frame  departures 

5 Statistics  Collection 


In  response  to  Which  line  do  you  want  to  change,  typing  the  code  letters  would 
cause  the  corresponding  questions  to  be  repeated. 


I 

i 
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2.1.  Introduction 

Several  variables  are  involved  in  a computer  communication  switching  system, 
and  each  one  of  them  may  affect  the  overall  performance.  A generic  configuration  of 
such  systems,  consists  of  communication  lines,  transmission  control  unit,  one  or  more 
processors,  core  memory  and  some  auxiliary  storage  devices  such  as  tapes,  drums,  and 
disks.  Since  the  number  of  parameters  is  large  and  each  can  vary  over  a vast  range, 
the  design  and  evaluation  of  such  systems  is  often  regarded  more  as  an  art  than  a 
science.  By  carefully  selecting  the  simplifying  assumptions,  a mathematical  model  can 
be  constructed  to  describe  the  influence  of  each  parameter.  In  this  chapter  we  shall 
concentrate  on  three  parameters;  number  of  processors,  processing  speed,  and 
required  storage.  The  most  common\y  used  criterion  for  performance  is  average  delay 
through  the  system.  The  other  criterion  used  is  thu  probability  that  delay  greater 
than  some  fixed  value. 

For  a communication  system  with  FCFS  (First-Come-First-Served)  discipline,  if 
the  distribution  of  message  lengths  is  the  same  over  all  input  lines,  an  integrated 
system  is  always  better  than  a segregated  one  with  same  total  input  rate  and  service 
rote.  Where  a segregrated  system  contains  two  or  more  separated  input  streams  and 
servers  with  each  independent  of  others.  But  for  different  distributions  of  message 
length,  the  integrated  system  may  be  worse.  These  properties  have  been  discussed  in 
[Ver  74].  In  the  system  discussed  here,  voice  and  data  are  of  quite  different  nature. 
For  a 10  millisecond  frame  and  a 8 Kbps  encoded  voice,  a call  of  3 minutes  implies 
transmission  of  18000*80  bits.  As  the  system  emulates  circuit  switching  for  voice,  this 
is  e(juivalent  to  a single  message  of  1260000  bits.  Since  the  maximum  data  pacKet 
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length,  is  only  2000  bits,  the  distributions  of  input  messages  are  substantially 
different.  Evaluation  of  this  kind  of  voice/data  integrated  system  was  first  done  by 
Kummerle  [Kum  741  by  Fischer  [Fis  75}  Both  concentrated  on  the  problem  of  the 
average  delay  seen  by  the  data  packet. 

An  outline  of  the  performance  model  follows.  We  will  use  Kendall’s  notation  for 
queueing  model.  For  the  first  two  parameters  *'V  refers  to  the  Markovian  character  of 
Poissonian  arrivals  and  exponential  service,  "O”  refers  the  deterministic  arrivals  and 
constant  service,  "GI"  refers  to  the  general  independent  distributions  of  inter-arrival 
time  and  service.  The  third  parameter  refers  to  the  number  of  servers  in  the  queueing 
system. 

General  Performance 

; The  integrated  switch  will  be  inter-connected  in  a communication  network, 

j termed  the  backbone  system.  Local  access  lines  will  tie  to  processors  in  the  backbone. 

{ In  the  backbone  system,  every  Class  I or  Class  II  message  is  assumed  to  reside  in  core 

j 

I memory  at  all  times  in  order  to  handle  the  large  amount  of  traffic  flow  and  to  satisfy 

the  short  delay  requirement.  Several  models  have  been  constructed  in  an  attempt  to 

I 

understand  the  stochastic  behavior  of  a data  packet  while  it  is  being  processed  in  the 
switch. 

j (i)  M/M/y  model.  There  are  c processors  in  the  system.  The  Class  I message 

I * 

' will  be  line  switched  and  be  assigned  a virtual  logical  channel.  A processor  can  handle 

i 

J 

p voice  channels.  Every  Class  I logical  channel  consumes  1/p  fraction  of  processing 
I power  of  a processor.  The  processing  power  which  is  free  to  handles  Class  11  packets 

it  now  y"(c-v/p),  where  v is  the  number  of  logical  channels  occupied  by  Class  1 

I 

1 2-2 

I 

1 
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messages.  This  kind  of  model  has  been  first  studied  by  Kummerle  using  an 
approximation  algorithm[Kum  74],  Then  Fischer  and  Harris  did  an  exact  analysis 
assuming  that  y,  a random  variable,  was  independent  from  one  frame  to  the  next  frame. 
Both  assumed  that  p-1.  In  the  present  case,  Class  I messages  have  a long  holding  time 
in  order  to  take  advantage  of  line-switching.  So  y is  highly  correlated  from  one  frame 

e^ 

to  another.  Later  Bhat[Bha  75]  did  an  analysis  of  multi-channel  queueing  system  with 
heterogeneous  classes,  where  the  number  of  simultaneous  equations  to  be  solved  is 
s(s-*-l)/2.  The  exact  analysis  is  given  in  the  next  section. 

Ih  our  experiment,  constant  Class  I traffic  is  assumed.  In  the  real  situation, 
ihtegrated  switch  in  local  loop  shall  take  the  responsibility  of  flow  control  to  keep  the 
backbone  system  from  getting  more  traffic  than  their  service  ability  of  Class  II 
packets.  Therefore,  in  the  following  models,  the  loading  factor,  which  is  the  ratio  of 
service  request  of  Class  II  traffic  over  the  corresponding  service  ability,  is  assumed  to 
be  constant. 

Performance  model  of  the  experiment 

(ii)  D*M/M/c  model.  In  this  model,  the  variation  of  the  system  loading  during  a 
frame  period  will  be  discussed.  Since  Class  I traffic  is  assumed  to  be  constant,  b tasks 
are  generated  every  d seconds.  The  homogeneous  Poisson  input  of  Class  II  messages 
generates  one  task.  The  service  time  required  by  each  task  of  either  class  has  the 
same  exponential  distribution.  The  system  will  appear  different  to  the  tasks  of  the  two 
different  streams.  The  0 stream  tasks  will  see  a less  crowded  system  than  the 
Poissonian  tasks  because  of  the  regularity  of  the  input  process  of  the  0 stream.  The 
results  will  show  that  the  difference  is  significant  at  low  traffic  intensities  and 
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t diminishes  as  the  traffic  intensity  increases.  The  influence  of  the  existence  of  the  D 

j stream  is  given  by  an  explicit  formula.  The  result  can  also  be  used  to  correct  the 

I 

I 

I simulation  results  when  the  statistics  of  the  simulation  is  gathered  at  some  specific 

I 

I point  in  the  frame  period. 

i ■ . . . • , 

Internal  Structure  of  Integrated  Switch 

The  above  two  models  discuss  the  total  number  of  data  packets  being  processed 

I 

I or  waiting  to  be  processed  in  the  system.  They  also  dictate  the  number  of  buffers 

j needed.  However,  no  information  about  the  structure  of  the  internal  task  queues  was 

I available.  The  following  models  try  to  attack  this  problem. 

Basically  there  are  two  types  of  integrated  switch  internal  structures, 
distributed  , and  centralized  . In  the  distributed  system,  each  processor  is  assigned  to 

t ■ ■ 

some  specific  task,  and  each  service  center  has  fixed  processing  power  to  process  its 
task..  While  in  the  centralized  system,  all  processing  power  is  concentrated  to  service 
I the  task  of  highest  priority.  Both  systems  will  be  discussed  in  the  models  that  follow. 

I 

(ill)  Open  Network  Model.  An  open  system  is  defined  to  be  one  in  which  the 
input  process  is  independent  of  the  output  process.  It  is  a good  model  for  a 
processing-bounded  system,  where  the  memory  is  assumed  to  be  very  large  but  the 
processing  power  is  limited.  In  this  model,  the  structure  and  the  waiting  time  of 
queues  are  analyzed.  The  distributed  system  is  modeled  as  an  open  queueing  network 
which  can  be  solved  by  techniques  described  in  [Bas  75]  for  exponential  service 
request.  The  centralized  system  is  modeled  as  an  FBfj  queueing  system  which  was 
' solved  by  Schrage[Sch67] 

(iv)  Closed  Network  Model.  In  a closed  network,  the  number  of  customers  in  a 
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tystcm  is  constant.  This  is  a good  model  for  storage-bounded  system,  where  an 
iiKoming  message  may  not  find  a buffer  and  has  to  be  neglected.  The  message  which 
it  neglected  by  the  node  is  not  lost  in  the  overall  communicatioa  Since  the  sender 
cannot  receive  an  acknowledgement  of  this  message,  it  will  send  the  message  again 
later.  But  the  message  does  suffer  an  extra  delay  because  of  the  finite  buffer 
capacity.  If  this  situation  happens  often,  the  delay  of  message  across  the  network  will 
depend  on  the  period  of  retransmission  of  a neglected  message.  The  closed  network 
models  are  constructed  by  adding  an  extra  queue,  which  pools  all  the  empty  buffers. 
Into  the  corresponding  open  network  models.  For  non-exponential  service  request  of 
task  queues,  the  approximation  techniques  described  in  a series  of  paper  by  Chandy  et 
al.  [Chan  75A,  Chan  75B,  Herz  75]  are  used. 
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2.2.  M/M/v  Queueing  Model 

2.2.1.  Introduction 


1 

1 


I 


Because  of  the  different  the  characteristic  of  data  and  voice,  the  integrated 
twitch  can  be  modeled  as  a queueing  system  withq  heterogeneous  classes  of 
customers.  So  far,  systems  of  this  type  have  not  been  studied  as  extensively  as  those 
of  homogeneous  customers.  When  service  rates  of  two  classes  are  the  same,  there  is 
no  need  to  distinguish  between  customer  types  once  a customer  joins  the  system. 
However,  when  the  service  rates  of  two  class  are  different,  the  number  of  the  states 
of  the  system  increases  tremendously.  This  is  fully  illustrated  in  Kotiah  and  Slater  [Kot 
73]  where  a two  server  system  of  this  type  has  been  analyzed.  In  the  integrated 
switch,  different  classes  of  messages  not  only  have  different  service  rates  but  also 
have  different  service  disciplines.  Voice  or  the  Class  I message  does  not  wait,  while 
data  or  Class  II  message  can  be  buffered.  The  system,  therefore,  becomes  very 
complicated.  Kummerle[Kum  74]  first  gave  an  approximation  algorithm  to  estimate  the 
average  number  of  packets  buffered  in  the  system.  Then  Fischer  and  Harris  [Fis  75] 
did  an  exact  analysis  assuming  that  the  size  of  Class  I traffic  is  independent  from  one 
frame  to  another.  Later  Bhat  and  Fischer  [Bhat  75]  used  another  approach  assuming 
that  two  classes  of  traffic  compete  with  each  other  for  s channels.  They  also  gave  an 
approximation  algorithm,  since  for  s-20  the  number  of  simultaneous  equations  is  210. 
The  approximation  algorithm  approaches  the  true  value  when  the  ratio  of  Class  II 
message  service  rate  to  Class  I message  service  rate  is  less  than  1.  When  this  ratio 
increases  the  approximation  may  differ  by  one  or  two  orders  of  magnitude.  In  the  real 
system  we  will  expect  this  ratio  to  be  of  the  order  of  hundreds. 
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There  are  c processors  in  the  system.  Each  Class  I logical  channel  will  subtract 
1/p  portion  of  processing  power  of  a processor.  There  is  a maximum  number  n of 
logical  channels  which  can  be  occupied  by  the  Class  I jobs.  When  a Class  I job  enters 
the  system,  if  the  number  of  existing  logical  channels  is  n,  the  job  will  be  lost  without 
being  serviced.  On  the  other  hand,  if  the  existing  logical  channels  is  less  than  n then 
the  job  will  create  a logical  channel  and  will  be  serviced.  The  request  process  of  Class 
I logical  channel  is  a Poisson  process  with  input  rate  The  holding  time  of  the 
logical  channel  of  Class  I is  exponentialiy  distributed  with  mean  l/pj.  Class  II 
messages  will  not  create  logical  channels,  but  come  in  by  packets.  The  input  process 
of  packets  is  assumed  to  be  Poissonian  with  input  rate  A packet  can  not  be 
serviced  by  more  than  one  processor  simultaneously.  The  service  time  of  a packet  is 
assumed  to  be  exponentially  distributed  with  mean  I/112  by  one  processor.  So  if  a 
packet  can  only  get  a fractional  power  of  a processor,  say  1/d,  then  the  service  time 
of  this  packet  will  be  d times  longer. 

2.2.2.  Algorithm 

The  queueing  system  is  a Markov  process  with  state  (i,j).  Where  i is  the  number 
of  Class  I logical  channel  and  j is  the  number  of  packets  in  the  system,  in  service  or 
waiting.  The  steady  state  probability  of  being  state  (i,j>  is  Pjj.  The  balance  equations 
for  M/M/y  model  is: 

(X,*\2)Po,0  •»*lPl,0*l*2Po,l 

(Xj+Xj+minf j,do)M2>Po,j  jP i ,j*^2Po,j-l  ''^o^kaPo.j*  1 l«0J>0 

<np,+X2+min<j,dn)M2)Pn,j  "^1^0-1, j*^2Pn,j-l*'"''’<j*^*'^n)<‘2Pn,j+l 
<na,4X2)Pn^0  ■^lPn-l,0*'"''’<^*‘^n>f2Pn,l 
(X|+i|i,*X2)Pj^0  ■^»Pi-l,0*<'*^>'*2Pi+l,0*'"'"<^''*i>»‘2Pi,l  0<l<n,j-0 
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(\|+i|i,+X2+min(j,dj)(i2)Pij  0<'<n.j>0 


2.1 


The  left-hand  side  of  the  above  equations  are  the  rate  of  probability  coming  into 
the  state  (i,j).  and  the  right-hand  sides  are  the  rate  of  probability  going  out  of  the 
state.  In  the  steady  state  the  rates  should  be  balance.  Where  dj-c-i/p  is  the 
processing  power  left  for  Class  11  packets  given  that  i Class  I logical  channel  exists  in 
the  system. 

Because  the  Class  I traffic  has  inherently  pre-emptive  priority,  so  for  fixed  n 
the  GOS  ( Grade'  Of  Service),  which  is  the  probability  that  a Class  I job  is  not  serviced 
because  no  logical  channel  available,  is  not  a function  of  Class  II  traffic  intensity.  GOS 
is  equal  to  Erlang  B loss  formula[SysK  60],  i.e., 

GOS(n>-(Xj/p,)"/n!/[.E  (Xj/Mj)j/j!]  2.2 

•nd  Pj-jC  Pij-(X,/M,)7i!/[^§Q(Xi/M,)Vk!]  2.3 

where  P|  is  the  probability  that  i logical  channels  exist  in  the  system.  Figure  1 
and  figure  2 shows  the  design  decision  of  n for  certain  given  GOS  value  and  the 
furKtion  of  GOS  with  respect  to  traffic  intensity  and  number  of  channel  n. 

Define: 


00  00 


logical  channels  exists  in  the  system.  Then  the  set  of  equation  2.1  can  be  re-write  as 
follows: 


[\  I +111 , ♦X2(  1 “z)+dj  ji2(  1 - 1 /2)]ni(z)-i  ji  1 Hi  - 1 (z)-\  I n i4l(^> 


00 


define  bj(z)  equal  to  the  right  hand  side  of  the  above  equations,  and 


. Uj(z)-X , ♦ip  1 ♦X2(  1 -z)+diP2<  1 - 1 /z) 
Up(z)-np , ♦X2<  1 -z)+d„M2^  1 1 /z) 


Osi<n 


2£ 
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•bove  equations  can  be  re-written  in  the  matrix  form  as  follows: 
AU)nUHl-l/z)b(z) 


where 


n(zHno(z)  njfz)  — n^Cz)]^ 
b<z)-[bo(2)  bj(z)  — b„(z)]^ 


ujz)  -X, 

-Ml  u/z)  -X, 


O 


-In,  u,(z)  -X, 


o 


. 

-nm  Un(z) 


The  above  equation  can  be  solved  by  conditions  that 

(I)  Z-1,  Hjd)-! 

(ii)  Jiitz)  is  analytical  in  the  region  Osz<l 

* • • 

Define  matrix  R(z)  with  (i,j)  elements  rjj(z) 
fj  j(zW-l)*'*’*(cofactor  of  the  (j,i)  element  of  matrix  A(z)) 

Then  R(z)A(z)-A(z)R(z)-determinant  of  A(z)-|A(z)| 

n(2)-|^(l-l/z)b<z)  2.7 

Theorem  1.:  lA(z)l*determinant  of  A(z)  has  exactly  n different  roots  in  (0,1),  and  Z“1  is 
also  a root. 

This  theorem  can  be  proved  similar  to  the  methods  of  [Mitr  68).  The  basic 
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argument  follows:  |A(1)|>0  is  trivial  for  the  row  summation  of  A is  zero.  Define 
g}(z)  as  the  principal  minor  of  order  i of  A(z).  |A(z)|  is  a polynomial  of  degree 
(2n-»-2):  using  standard  techniques,  such  as  counting  the  number  of  sign  changes 
in  the  sequence  gj(z),  i«0,l,2_,n  at  z»0  and  z«l,  one  can  determine  the  number 
of  zeros  of  |A(z)(  in  (0,1).  If  Zj^'^  is  the  jth  root  of  gj(z),  then  for  a special 
property  of  A(z)  that  consecutive  gj+i(Zj^'h  will  have  difference  signs.  So 
roots  of  gj+i(z),  will  be  in  (Zj_j^'^,Zj^'^)  interval  for  j-0,l,_,i.  With 
conditions  that  Zg-0,  only  one  root  of  ggtz)  is  in  (0,1),  and  1 is  the  root  of 
«n<2)  •■|A(z)|,  the  proof  can  be  carried  by  the  above  interval  partition  procedure. 
The  same  procedure  is  also  done  numerically.  e 

Theorem  g:  R(Zj)  has  rank  one  for  Zj,  a root  of  |A{z)l  in  (0,1]. 

The  proof  is  very  tedious  but  straightforward.  Basically  the  proof  is  calculating 
the  following  formula 

'’jK<*i^'’ml<^i)-'’jl<^i>'’mk<Zi) 

Each  element  in  the  above  formula  is  a minor  of  A(z).  By  expanding  the 
determinants,  term  by  term  they  will  combine  with  each  other  to  a form  of 
pfZj)- |A(Zj)|,  so  the  formula  can  be  proved  equal  to  zero  for  any  j,k,m,l.  So  the 
matrix  R is  of  rank  1 for  z,  a root  of  A(z).  e 

Because  R(z)  has  rank  1,  so  we  can  choose  arbitrary  row  of  R(z)  as  r(z).  By  the 
above  theorems,  we  have  (n-fl)  equations: 

r(Zj)b(Z|)-0  2-8 

for  i-l,2,...,n 

ar>d 
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r(l)b(l)-^A(z)|2.i  2.9 

j The  unknown  variables  are  all  P^jj  for  Osj<dj,  ng  are  defined  as  the  number  of 

I • w 

these  variables.  Where  P]|i”Pi,j/jQ  ^ij-  While  number  of  equations  in  2.4  for  all 

j Osj<dj  is  ng-fn+l).  So  together  with  the  above  (n+1)  equations,  Pj|j  can  be  solved. 

i The  mean  number  of  packets  given  that  there  are  i logical  channels,  Rj’d),  can  be 

solved  by  following  method. 

As  equations  2.4  and  2.6,  A(2)R(2)-(l-l/z)b{z) 

TaKe  derivative  with  respect  to  z of  the  above  equation 

I A’(z)n(z)+A(z)n’(z)-l/22-b(z)+<l-l/z)b’(2)  . 2.10 

I then 

i n(z)-j^[l/22-b(zHl-l/z)b(z)-A’(z)n(z)]  2.11 

I 

i Take  limit  as  z approach  to  1, 

I n’(l)-l/[^|A(z)|2.i}{R’(z)[b(z)/22-A’{z)n(2)] 

I ♦R(2)-[-2b(z)/z3+2b(z)/z2-A"(2)n(z)-A(z)n’(z)]}2.i  2.12 

I 

) 

Then 

R ( 1 )-{r’(  1 )[b{  1 )-A  ( 1 )n(  1 )]+R(  I )[-2b(  1 )+2b’(  1 )-a"(  1 )n(  1 )-a’(  1 )n{  1 )]l/a  j 

• d ” 

. • where 

• m 

A(l)  and  Ad)  are  diagonal  matrix  with  ith  element  (-Xg-Kljiig)  and  (-ZdjPg) 
* respectively.  Then  Rj  d)  can  be  written  in  the  following  form: 


R;d)-l/.,  {4i^ag>.E  g.n/d)) 

n , 

where  rjj  (dCbjdKXj-djajl 
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'’nj(l>t-2bj(l)+2b/(l)*2dja2] 

All  the  above  parameters  are  known,  njd)  the  mean  number  of  packets  for  I 
logical  channel  can  then  be  calculated. 

2.2.3.  Numerical  Consideration 


f 

\ 


f 


P 

\ 


Number  of  simultaneous  equations  has  to  be  solved  by  the  above  algorithm 
depends  on  p and  c.  For  p-1,  the  special  case,  the  system  is  similar  to  system 

j analyzed  in  [Bhat  75].  The  number  of  simultaneous  equations  is  (n'«-l)n/2.  But  for  c-1, 

! 

I 

{ 03  equal  to  n,  the  maximum  number  of  logical  channels. 

I 

! The  algorithm  discussed  in  the  last  section  will  be  numerically  stable  for  |ii  arid 

^2  •r'  the  same  order  of  magnitude.  As  the  ratio  m/fz  increases,  the  algorithm 
becomes  less  stable.  Double  precision,  or  even  quadruple  precision  if  available,  is 
suggested  for  in  the  order  of  thousands.  Calculating  the  roots  of  |A(z)|  in  (0,1) 
I and  calculating  r(Zj)  are  the  two  most  sensitive  steps.  Principally,  |A(z)|  is  a polynomial 

! of  z of  order  2n«-2.  The  coefficients  of  this  polynomial  can  be  calculated,  and  the 

i 

j polynomial  solved  using  a standard  package.  This  method  is  not  recommended  for 

I large  reason  is  that  all  the  variables  of  |A(z)|  are  in  the  form  of  a large 

number  plus  or  minus  a small  number.  Multiplication  and  division  of  such  numbers  are 
I numerically  unstable.  For  example  (1.001*1.002*1.003)  can  not  be  distinguished  from 

I (1.002)^  for  a machine  with  20  bits  for  mantissa.  More  accurate  but  quite  costly  is 

I substituting  the  iterative  z into  lA(z)|  directly  and  calculating  the  determinant. 

I 

i Calculation  of  r(Zj),  principally,  is  calculating  the  principal  minor,  which  is  the 

‘1  determinant  of  a principal  submatrix  of  A(zj).  The  argument  of  instability  is  the  same. 

I 

I Because  R(Z|)  has  rank  1,  there  are  a lot  of  redundancy  in  RtZj).  The  right  choice  of  a 

I 

i 

> 2-12 

i 

I 

__  - ‘ Jdfrtn  I 
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row  will  yield  a much  better  result  than  others.  As  a rule  of  thumb,  if  the  computation 
of  |A(Zj)|  is  backward  deration  from  n to  0,  then  the  computation  of  r(Zj)  should  be 
backward  iteration  from  0 to  n,  and  vice  versa.  The  backward  iteration  algorithm  is 
used  for  its  stability.  The  details  of  the  error  analysis  in  backward  iteration  can  be 
found  in  [Jone  74]. 


2.2.4.  APDroximation  Algorithm 


The  approximation  algorithm  of  [Bhat  75]  fail  for  large  Pi/mj*  which  is  the 
practical  situation.  We  will  give  here  a more  complicated  but  much  more  accurate 
approximation  for  the  special  case  where  c-1.  From  the  numerical  result  of  above 
analysis,  it  shows  that  the  average  number  of  packets  for  Pi/P2  heavily  loaded 
system  are  not  sensitive  to  c,  the  number  of  processor  in  the  system.  Where  the 
heavily  loaded  system  is  defined  that  the  input  rate  of  class  II  traffic  is  greater  than  c- 
n/p.  It  is  equivalent  to  say  that  for  segregated  systems  with  same  total  processing 
power  with  the  same  degree  of  service  for  Class  I system,  then  the  Class  II  system 


will  have  traffic  intensity  greater  than  one. 

C-l,  di-l-i/p,  and  gi-.5jjPi.jA5)  ^i 
Then  equation  2.4  becomes 


-<Al+'Pl)gi+'P|ai-l*^igi+l— ^Z+djPzd-Poii^ 

Assuming  that  PQij-l/d+gj)  and  substituting  to  the  above  equations,  we  will  get 
g|.  We  call  this  the  conditional  mean  approximatioa 


2.23.  Comparison  of  Results 


For  constant  k|/^,  arwJ  X2/P21  average  number  of  packets  increases  rapidly 


1 


2-13 


1 


Carnegie-Mellon  University  Report 


Juno  26,  1976 
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! 

I 

i 


when  112/M1  increases.  This  behavior  is  also  shown  in  [Bhat  75].  There  is  no  channel 
reserved  only  for  data  packets  in  that  article.  In  our  model,  when  the  input  rate  of 
Class  11  packets  is  greater  than  d^,  which  is  the  worst  case  processing  power  for  Class 
II  traffic,  similar  behavior  is  shown.  Otherwise,  the  system  is  not  very  sensitive  to 
Notice  that  when  dj>X2  tor  some  i,  the  average  number  of  packets  will  be  quite 
large  when  there  are  i Class  I logical  channel.  The  reason  is  that  once  the  system 
goes  into  this  state,  the  input  rate  is  greater  than  the  processing  rate,  so  packets 
accumulate.  Until  sometimes  later,  the  system  can  give  more  processing  power  to  the 
Class  II  packets  for  the  releasing  of  some  Class  I logical  channels,  then  the  number  of 
packets  will  decrease.  The  integrated  switch  has  a large  variation  of  number  of 
packets  in  the  system.  This  is  in  some  sense  a trade  off  between  the  usage  of 
processing  power  and  memory. 

Table  1 is  the  average  number  of  Class  II  packets  in  system  given  that  i Class  I 

logical  channels  exist.  For  different  c,  the  number  of  servers,  nj(l)  differs  very  little 

from  each  other  on  overload  states,  which  are  states  djSX2.  Because  of  this  character, 

although  the  approximation  algorithm  is  based  on  uni-server  system,  mean  number  for 

overload  states  should  be  a good  approximation.  Another  very  impot  tant  character  is 

that  the  conditional  mean  number  of  packets  vafies  a lot  for  different  i.  Even  with 

rather  small  mean  waiting  length,  the  probability  of  overflow,  which  is  defined  as  the 

» 

number  of  packets  in  system  exceeds  its  buffer  space,  is  quite  high. 

Table  2 and  3 are  comparison  to  the  results  of  Kummerle’s  and  Fischer’s 
respectively.  In  Kummerle's  approximation,  the  service  scheme  is  the  same  as  ours, 
but  the  service  time  of  packets  is  constant  rather  than  exponential.  Both  Kummerle’s 
arKj  our  results  show  the  sharp  increase  of  the  conditional  mean  in  overloaded  states. 
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In  table  2 we  also  note  that  the  mean  increases  as  increases  although  ^|/Pi  and 
X2/P2  •ra  hept  constant.  In  Fischer  and  Harris’s  TN  6-75  OCEC  report,  they  fail  to 
show  this  characteristics.  In  their  later  TC  21-75  OCEC  report,  the  new  model  has  this 
characteristics,  but  they  only  tend  to  give  the  total  mean  waiting  time  while  as  we 
stated  before  is  not  enough  to  describe  the  whole  system.  The  TC  21-75  OCEC  report 
is  the  special  condition  where  P"1  and  cp^n  of  our  analysis.  The  numerical  results  of 
[Bhat75]  , which  does  not  allow  the  Class  H packet  preemptive  the  Class  I traffic,  are 
compared  here.  The  blocking  probability  of  [Bhat75]  is  much  higher  than  our  result 
because  in  tBhat75]  no  priority  is  p"  an  to  Gass  I traffic,  for  the  same  reason  it  has 
much  shorter  waiting  line  in  the  queue. 


I 
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Table  1 Comparison  of  exact  solution  and  approximation 


^ cp-15s  n«10;  \j/»i,»5.05  X2/cM2“7/15; 


Hjd) 

Pi 

C»1 

c-3 

c-5 

Approx. 

1-0 

.0068315 

0.8881841 

1.5899381 

2.4360016 

.876 

i-  1 

.0341575 

1.0399755 

1.7091119 

2.5267855 

1.002 

1-  2 

.0853938 

1.2894188 

1.9175224 

2.6971034 

1.170 

i-3 

.1423231 

1.7669488 

2.3417091 

3.0699985 

1.405 

I-  4 

.1779038 

2.7981681 

3.3011327 

4.0220692 

1.762 

I- 5 

.1779038 

5.1520923 

5.5606589 

6.2629728 

2.399 

1-  6 

.1482532 

10.4236670 

10.7955900 

. 11.3819070 

4.188 

i-  7 

.1058951 

21.0693230 

21.4009170 

21.9430840 

11.637 

i-8 

.0661845 

38.8442910 

39.1415930 

39.6471080 

29.404 

i-  9 

.0367691 

61.7339510 

62.0130820 

62.4995080 

53.226 

i-10 

.0183846 

81.8106690 

82.0841770 

82.5715340 

73.894 

Total  mean 

11.93888501 

12.38936477 

13.05113306 

8.195 

100 


0l  234  5 6.  789  10 


NUMBER  OF  REAL  TIME  CHANNELS 
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Table  2 Comparsion  of  Kutnmerle’s  approximation 

for  15  channels;  Xj/»ij-5.0;  \2hz^7ll5\ 

Kummerle’s  Approx.  Exact  Calculation  Cond  Mean  Approx. 


Aa/fi 

10 

100 

10 

100 

10 

100 

0 

.9375 

.9375 

.92832 

.88818 

.891 

.876 

1 

1.0714 

1. 07 1 4 

1.10053 

1.03998 

1.022 

1.002 

2 

1.2500 

1.2500 

1.36056 

1.28942 

1.202 

1.170 

3 

1.5000 

1.5000 

1.76762 

1.76695 

1.469 

1.405 

A 

1.8750 

1.8750 

2.40928 

2.79817 

1.908 

1.762 

5 

25000 

2.5000 

3.39682 

5.15209 

2.696 

2.399 

6 

3.7500 

3.7500 

4.84210 

10.42367 

4.072 

4.188 

7 

7.5000 

7.5000 

6.81077 

21.06932 

6.175 

11.637 

8 

8.4999 

8.4999 

9.25161 

38.84429 

8.888 

29.404 

9 

11.7024 

27.7738 

11.89144 

61.73395 

11.814 

53.226 

10 

18.0428 

73.3285 

14.03198 

81.81067 

14.144 

73.894 
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Table  3 Comparsion  of  Bhat’s  model 


for  2 channels,  pj^-0.5, 


Bhat’s  Model 


Cond  Mean  Approx. 


«c 

PB 

Et02] 

E,[Q2] 

PB 

• EJOj] 

.0005 

.2461 

.5333 

.5001 

.1429 

.3335 

.005 

.2461 

.5336 

.5005 

.1429 

.3352 

.05 

.2462 

.5368 

.504 

.1429 

.3517 

5 

.248 

.5618 

.5505 

.1429 

.4867 

1 

.25 

.5833 

3833 

.1429 

.6026 

5 

.2574 

.6945 

.6605 

.1429 

1.205 

25 

.2629 

1.046 

.6965 

.1429 

3.071 

100 

.2646 

2.212 

.705 

.1429 

8.585 

500 

.2651 

8.34 

.71 

.1429 

35.835 

1000 

.2652 

15.994 

.71 

.1429 

69.552 

5000 

.2652 

77.219 

.705 

.1429 

33a839 
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2.3.  D^M/M/c  Queueing  Model 

I 

The  integrated  switching  system  is  modeled  as  a queueing  system  whose  inputs 
are  composed  of  two  independent  streams.  The  concept  of  a piecewise  MarKov 
process  is  used  to  analyze  such  a queueing  system.  One  of  the  streams  (which  is 
corresponding  to  the  flow  of  data  packets)  is  Poissonian  with  intensity  X.,  and  the  other 
(which  is  corresponding  to  the  constant  level  of  voice  slots)  is  assumed  to  be  a 
deterministic  renewal  process  with  intensity  r and  batch  of  size  b.  The  service  times 
of  all  the  tasks  are  independent  and  identically  distributed  according  to  an  exponential 
distribution  with  mean  This  kind  of  system  has  been  analyzed  by  Kuczura[KUCZ 
72]  as  GI+M/M/c  case  with  c-l  and  bo.  We  will  generalize  the  result  to  arbitrary  c and 
a batch  input  of  magnitude  b.  The  GI  stream  is  assumed  to  be  a deterministic  process, 
for  every  d second  there  will  be  a batch  of  b tasks. 

The  state  of  the  system  (number  of  tasks  waiting  and  in  service  ) seen  by  a task 
will  generally  differ  from  that  of  the  0 stream  tasks.  Consequently,  in  a system  where 
tasks  wait  for  service  such  as  O+M/M/c  queue,  the  service  received  by  the  two  types 
of  customer  will  differ.  For  non-batch  and  first-come-first-served  system,  the  0 
stream  tasks  always  receive  better  service  than  the  Poissonian  tasks  because  of  its 
regularity  of  input  process.  While  significant  at  low  traffic  intensities,  this  advantage 
diminishes  as  the  traffic  intensity  increases. 

Let  Y(t)  be  the  number  of  task  in  the  system  (those  waiting  and  in  service)  at 
time  t.  Since  the  D stream  is  a renewal  process  and  Y(t)  is  Markovian  between  any 
two  consecutive  arrival  epochs  of  the  tasks,  {Y(t),  tzO)  is  a piecewise  Markov  process 
with  state  space{0,l,2,...}.  The  degeneration  points  are  the  arrival  epochs  of  the  D 
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tasks.  The  Markov  process  operating  with  the  segments  is  a birth-death  process  with 
birth  rate  X and  death  rate  p,  the  same  for  all  segments.  The  transition  probability  can 
be  calculated  numerically  by  following  algorithm: 

p*(t)-Ap(t)  3.1 

where  p(t)  is  the  probability  column  vector  Pj(t)-Prob[Y(t)»i]  and  A is  the 


infinitesimal  generator. 


-X.  M ^ 

» 1 


o 


k,  >^1 


where  pj  ■ min(i,  c)*p  is  the  processing  rate  of  system  when  i jobs  in  the 
system. 

p{t)-exp(At)p(0)-[l+(At)*(At)2/2!*(At)3/3!+_]p(0) 

P(t)  is  the  transition  probability  matrix  ■ transpose  of  exp(At). 

P|  jftMransition  probability  from  state  i to  state  j during  a time  interval  t. 

(Pj)  ^ stationary  distribution  of  the  Markov  chain  imbedded  at  points 
immediately  preceding  a D task  arrivals.  If  ry  is  the  one-step  transition  probability 


from  state  i to  state  j,  then 


'’IJ“Pi*b,jW> 


i«i“0»lf2.3i. 


where  d is  the  inter-arrival  time  of  the  0 tasks.  The  distribution  (pj)  satisfies 

the  Chapman-Kolmogorov  equations. 

00 

Oi-E  D:r!:.  i-0.1.2.~  3.3 


subject  to  the  normalization  condition 
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il-r*  3.4 

it  the  stationary  distribution  of  {pj}. 

L«t  {qp  be  the  stationary  distribution  of  the  state  seen  by  an  arbitrary,  arriving 
Poisson  tasKs.  Using  rate  conservative  principle  we  can  write: 

. i»Pj+\qj+min(i,  c)j»qi-min(i+l,  c)»iqj.^i+\qj_i*irpj.b 
where  the  left-hand  is  the  asymptotic  rate  of  transition  leaving  state  i and  the 
right-hand  side  is  the  asymptotic  rate  of  transition  coming  to  state  L No  simple  closed 
form  solution  is  available.  It  can  be  calculated  numerically  without  much  difficulty.  For 
b-1,  the  closed  form  solution  is  available  as  follow: 

Uj+^qj-Cj+Dpqj+l  OSj<c  35 

rpj+\qj-C(jqj+i  j2e  3.6 

The  left-hand  side  is  the  asymptotic  rate  of  transition  from  j to  j-t-l  and  the 
right-hand  side  is  the  asymptotic  rate  of  transition  from  j+l  to  j.  Multiplying  both 
sides  by  ui  and  summing  over  all  ],  we  obtain  the  following  relation  between  p(u)  and 


i»p(u)+Xq(u)-c>i[q(u)- E ( 1 -i/c)qju']/u 

|«Q 


q(u)-[  ( 1 -i/c)qju'  -t-ueptu)]/!  1 -pu) 


where  #«r/cp  and  p-X/cp.  Applying  the  normalization  condition  We^pil 


(l-‘/c)qj-l-a-» 

i«o 

along  with  the  following  (c-1)  equations. 


(i+Dliqj^l-Xqj-rpj 

C|^|/)2>-Rc-1  solved. 

qj-[»Pi-i^xqj-i]/cp 


O^iSc-2 


t 
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3.12 


If  1 1 is  the  mean  of  the  distribution  {pj}  and  I2  is  the  mean  of  the  distribution 


{qj),  then  l|-p’(l),  and  l^-q’d). 
c-1 

q’(l K E ( 1 -i/c)iqi+»+ap’(  1 )]/( l-p)+p/(  1-p) 

1*0 

e-1 

I2-CP+.^,  (l-i/c)iq;]/(l-pW(l*l,)/(l-p) 

I"!  ' 


3.13 

ai4 


Note  that  the  first  term  on  the  right-hand  side  of  the  above  equation  is  the 
mean  number  of  tasks  in  the  system  M/M/c  queue  with  traffic  intensity  p.  Hence,  the 
second  term  may  be  considered  to  be  an  increase  in  the  mean  number  due  to  the 
presence  of  the  0 tasks. 

This  D+M/M/c  model  describes  the  effect  on  the  system  by  changing  the  level  of 
the  deterministic  stream  of  class  1 tasks  in  the  multi-server  environment.  If  the 
statistics  are  gathered  at  the  end  of  each  frame,  then  {pj}  will  be  calculated.  On  the 
other  hand,  if  the  statistics  gathering  are  triggered  by  the  input  of  Class  II  packets  or 
by  some  random  snap  shot,  {qj}  will  be  calculated.  From  another  point  of  view,  if  both 
{Pj}  and  {qj}  are  gathered,  p,  the  service  rate,  can  be  estimated. 
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2.4.  Internal  Structure  of  Integrated  Switch 

In  this  section,  the  internal  queueing  structure  of  the  integrated  switch  is 
discussed.  The  model  and  results  heavily  depends  on  the  task  decomposition  and  the 
processor -memory  structure  of  the  system.  Barbacci  and  Oakly  [Barb  76]  give  a detail 
tdsk  decomposition  of  an  integrated  switch.  That  work  is  primarily  designed  for  C.mmp. 
The  task  decomposition  scheme  is  assumed  to  be  generalized  for  any  system.  The 
other  factor  is  the  PMS  (Processor  Memory  Switch)  structure  of  the  system.  There  are 
various  PMS  structures  for  an  integrated  switch.  We  will  partition  them  into  two 
categories,  distributed  system  and  centralized  system.  The  distributed  system  implies 
that  the  processing  power  is  restricted  to  do  some  specific  tasks.  A typical  well- 
krK)wn  example  of  a distributed  system  is  a computer  with  a front-end  processor.  The 
main  processor  or  the  front-end  processor  in  generally  will  not  do  the  tasks  pre- 
assigned to  the  other  even  it  is  idle  and  capable.  A more  complicated  system  is  CM*  of 
Carnegie-Mellon  University  . Each  processor  has  its  own  local  memory  in  which  some 
but  not  all  of  the  application  programs  are  stored.  Although  processors  can  fetch  the 
application  program  stored  in  global  memory  or  local  memory  of  other  processors,  it 
will  suffer  extra  delay  through  the  memory  mapping  mechanism.  So  in  the  distributed 
system,  each  processor  is  assumed  to  be  assigned  to  some  specific  application 
programs,  and  it  will  only  execute  tasks  corresponding  to  it.  In  the  centralized  system, 
the  processor  v.'iil  ^xev..<*9  any  tasks.  The  centralized  system  has  a large 
homogeneous  memory  which  stores  all  the  buffers  and  application  programs.  The 
ordinary  uni-processor  system  is  the  typical  example.  The  processor(s)  can  be  located 
to  do  any  application  program  at  any  time  without  suffering  extra  delay.  The 
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processing  power  of  the  system  is  centralized  For  some  instant,  all  the  processor(s) 
may  execute  the  same  application  program,  while  this  situation  can  never  happen  in 
the  distributed  system. 

A networK  of  queues  is  used  to  model  the  distributed  integrated  switch. 
Networks  of  queues  have  been  widely  studied  as  a multi-programmed  and  time-shared 
computer  systems.  Jackson  [Jack  63]  and  Gordon  and  Newell  [Gord  67]  developed  the 
equilibrium  distribution  of  states  of  a class  of  general  networks.  In  particular,  Gordon 
and  Newell  make  clear  the  product  form  of  the  solution  of  the  balance  equations 
describing  the  steady  state  of  the  model.  In  these  models  the  service  center  can  be 
connected  in  any  arbitrary  fashion.  A customer  leaving  a service  center  simply  choose 
the  next  service  center  according  to  a set  of  branching  probability  for  the  center 
being  left.  Jackson's  model  also  allows  for  the  arrival  and  departure  of  customer 
outside  the  system.  There  is  a big  constraint  in  these  models  that  all  the  service 
request  must  be  exponentially  distributed.  Baskett  et  al  [Bask  75]  generalized  the 
networks  containing  several  class  of  customers  and  non-exponential  service  time  for 
special  service  discipline.  The  result  also  has  the  product  form.  In  a series  of  paper 
of  Chandy  et  al  [Cha  75],  an  approximation  algorithm  to  solve  a more  general  non- 
exponential  service  closed  network  is  discussed.  The  first  iteration  of  the  algorithm  is 
the  solution  for  exponential  service  network.  These  techniques  are  used  to  solve  the 
distributed  system. 

For  a centralized  integrated  switch,  models  of  FBjyj  queue  system  and  a M/M/l/\j 
queue  system  are  used.  Where  FB|\j  queueing  system  consists  of  a service  center  and 
queues  of  different  service  time.  When  a customer  finishes  the  service  in  ith  queue,  it 
Will  branch  to  the  (i'»'l)st  queue  with  some  fixed  probability.  M/G/l/Xj  queueing  is 
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used  to  modeled  the  closed  network  model  where  the  input  rate,  Xj,  of  the  queue 
dependents  on  the  state  of  the  queue  which  is  the  number  of  customer  in  the  system. 
At  stated  before,  open  network  model  is  used  for  analyzing  the  service  structure  of 
the  system  and  is  appropriate  for  processing-bounded  system.  A closed  network 
model  is  used  for  buffer  contention  and  is  appropriate  for  storage-bounded  system. 

2.4.1.  Open  Network  Model 

From  theorem  6.3  of  [Jack  63]  for  constant  arrival  rate  of  a system,  the 
equilibrium  probability  distribution  of  queue  length  at  the  individual  centers  are 
independent  and  also  each  of  these  distribution  is  identical  with  that  for  a isolated 
queueing  system  with  some  equilibrium  input  rate. 

The  routing-generating  process  is  specified  by  a set  of  parameters, 

R-{r(m,n)|m<[0,M]r<[  1 ,M+ 1 ]} 

where  M is  the  total  number  of  service  center.  The  interpretation  in  that  for  m 
and  n <[1,M];  (i)  the  probability  is  r(0,n)  that  Center  n will  be  first  on  a routing,  (ii)  the 
probability  is  r(m,M+l)  that  if  Center  m is  ith  on  a routing,  then  this  ith  element  is  the 
last  one,  (iii)  the  probability  is  r<m,n)  that  if  Centr  r m is  ith  on  a routing,  then  there  is 
a (i+Dst  element  and  it  is  Center  n. 

The  set  of  equations 
M 

•(n)»r(0,n)+  C e(m)r(m,n),  n([l,M]  4.1 

has  a unique  solution  {e(n)|n<[l,M]}i  and  the  e(n)  are  ail  non-negative. 

• The  task  decomposition  scheme  of  the  integrated  switch,  as  Fig.l,  is  a tree 
structure,  e(i)  can  be  solved  very  easily.  The  transformation  from  the  distribution 
system  to  the  centralized  system  is  as  in  the  figure: 
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T|->the  service  time  of  the  ith  tasK  queue;  the  length  of  processing  which  a 
customer  request  in  the  ith  queue  of  the  distributed  system. 

TjMhe  ith  simple  processing  time;  the  length  of  processing  which  a customer 


requests  in  the  ith  queue  of  the  centralized  system. 


I 


Similar  formula  can  be  written  if  the  task  decomposition  scheme  is  modified, 
and  e(0)-l,  e(l)>b|,  e(2)-b2,  e(3)-b2b4,  e(4)«b2b5,  e(5)-b2b4bg,  e(6)«b2b4b2, 
because  the  physical  meaning  of  e(i)  are  the  net  input  rate  of  the  ith  queue. 


Define  Wf,’-the  time  in  system  at  the  completion  of  the  nth  simple  processing 

time. 

This  FB|y|  queueing  system  was  solved  by  Schrage[Schr  67^ 
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prw  M ^(Shh)  F(T*)42 

^^"^‘2Ci-^e(l/*h)JIi"  xe(u,);  rFXlTiOT 

where  Tp  > the  nth  simple  processing  time;  the  length  of  processing  which  a job 
in  the  nth  queue  receives. 

Sp«T|’+T2’+~*Tp’  service  request  given  that  the  customer  returned  to  the 
system  at  least  n times. 

Up>Sp  if  the  customer  returned  to  the  system  at  least  n-1  items, 

-T|’+T2’+-.+T|^’  if  the  customer  returned  to  the  system  only  k-1  times,  k<n. 

A^-xn  b’j 

Defined  Sq'Uq-O,  and  b’Q-1,  then  Up-Up.j+b’p.jTp’ 


Here  exponential  service  time  of  each  task  is  assumed. 

Define  Pi"Xe(i)/(pjCj)  is  the  equivalent  traffic  intensity  of  service  center  i. 

Where  Cj  is  the  processing  power  in  service  center  I and  the  service  request  at 
service  center  i is  exponentially  distributed  with  mean  1/pj. 

Network  R is  fixed  for  some  fixed  task  decomposition  scheme,  so  does  e(i).  The 
average  length  of  queue  i will  be: 

r- 

E[li]"Aj/(l-Pi)  ^ 

and  Vartlj]-pi/(l-pi)2  4.4 

The  total  number  of  customers  in  the  system  I, 

M M 

E[l]-EE[lj]^Epi/(l-Pi)  4.5 

MM, 

and  Var[l]«.E  Var[lj]-.E  pj/d-pj)^  4.6 

(•I  ' 

The  optimum  way  to  distribute  the  processing  power  depends  on  the  criterion 


used.  Here  several  examples  are  given: 


I 
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If  the  criterion  is  to  minimize  C I:,  that  is  to  minimize  the  average  number  of 

n 

packets  in  the  system,  with  the  condition  that  C Cj*constant. 

1*1 

n n 

minimize  given  that 

It  is  a transform  of  the  iink  capacity  assignment  problem  studied  by 

Kleinrock[Klei  63].  The  optimize  way  is  so  called  the  square  root  assignment. 

Let  Cj*processing  power  assigned  to  service  center  i. 

M 

ZT  Cj-C«total  processing  power  of  system  -constant. 

‘fife  problem  becomes: 

M M 

Minimize  E kjApjCj-X:)  given  constraint  E Q-C. 

1-1  ' ' ' ' I-l  ' 

The  optimum  assigning  becomes 

Cj-Xj/|ij+e^(Xj/pj)^/2 

M M 

where  cC-(C-E  (Xj/pj))/E  (Xj/p;)^  ” 
i-l  ' ' 1-1  ' ' 

Other  criterion,  like  minimizing  the  kth  moment  of  Ij,  or  the  minimax  criterion  can 
be  used  as  referred  to  [Meiz  72}. 

Hero  gives  a numerical  example  to  calculate  the  waiting  time  of  a centralized 
system  and  a distributed  system  given  that  the  processing  capacity  assignment  is 

a 

minimizing  the  average  number  of  packets  in  system. 

Suppose  C-5,  Xq-1.0,  bj-.4,  b2~.6,  bg-.fl,  b4-.4,  b5-.2,  bg-.S,  by-.5,  TQ._Ty  are 
exponentially  distributed,  i.e.  En'^]-2- (E[T])^.  arKl  EtTQ)-E[T|]-E[T4]-E[T5]-1.0, 
E[T2]-E[T3]-E[T,]-2.0. 

From  the  task  decomposition  scheme,  e(0)-1.0,  e(l)-.4,  e(3)-.24,  e(4)-.12, 

e(5)-.12,  e{6)-.12, 

6 6 

ot-C-E  e(i)T:/.C  [e(i)T:]l  /2^  4 

1-0  ' t-0 

Waiting  Wj-l/(>iCj-Xi)-l/MXj/Ti]i/2) 

Wo-2.5,  W,-3.95,  W2-4.56,  W3-7.22,  W4-7.22,  W5-7.22  W4-10.20{ 


I 


i 
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! 

For  e«'')tralized  system,  we  can  calculate  the  bj*  according  to  the  tasK 
decomposition  graph.  b|’>1.0,  b2V36,  b3V6667, 

E[t/]-.2,  E[T2>.32,  E[T3  1-.2,  E[T;1-.3, 

Err,’2]-.08,  E[T2’2]-.1664,  E[T32]-.0672,  E[T4  2]-.14, 

E[S,]-.2,  E[S2]-.52,  E[S3]-.72, 

E[U,  ]-.2,  E[U2]-.52,  E[U3]-.592,  E[U4]-.792, 

E[U,2>.08,  E[U22>.3744,  E[U32]-.458,  E[U42]-.7573, 

From  the  formula  of  FB|yj  system  waiting  time  can  be  calculated; 

Wj-.5042,  W2-.974O,  W3-2.54,  W4-6.525, 

2.4.2.  Close  Network  Model 

. 1 

' There  are  a finite  number  of  buffers  in  the  system.  If  an  incoming  packet  can 
not  be  located  with  a buffer,  it  is  neglected  by  the  switch.  Because  of  the 
acknowledgement  feature  of  the  communication  network,  this  packet  will  be  sent  again. 

This  packet  is  not  lost  but  it  does  suffer  an  extra  delay.  In  this  section  the  probability 
that  this  situation  happens  is  calculated.  The  same  queueing  network  of  the  last 
section  is  studied  except  for  an  extra  queue  for  pooling  the  unoccupied  buffers. 

Centralized  System 

When  the  total  service  time  of  a customer  in  the  integrated  switch  is  exponential 
distributed,  the  probability  of  i customer  in  switch  is  irrelevant  to  their  service 
discipline  as  long  as  the  service  discipline  is  independent  of  the  service  request. 

There  are  n buffers  and  c servers  in  the  switch.  Let  the  state,  i,  be  the  number  of 
buffer  currently  occupied,  and  Pj(t)  be  the  probability  of  state  I at  time  t.  Then  the 
backward  Kolmogorov  differential  equations  can  be  written  as: 
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^Po<t)-APo(t)*pP,(t)  4.7 

^Pj(t)-XPj.i(tHX+ip)Pi(tHi*l)|iPj+j(t)  i<c  4.8 

^Pj(t)-XPj.i(t)-<X+cp)Pi{t)*CMPj^l(t)  n>iic  4.9 


Define  the  steady  state  probability  Pj-iimit  of  Pj(t)  as  t approach  to  infinite.  The 
left  side  of  the  above  equations  become  zero.  The  above  equations  can  be  solved 
easily: 

Pj-X'/OVVPo  i<c  4.11 

Pj-X«/(c!p<=)-(X/c»i)'"'Po  iic  4.12 

N 

with  .^Pi“l, 

P^  is  the  probability  that  a customer  comes  in  and  is  neglected  for  the  lack  of 
available  buffer. 

Define  Sp-probability  that  a customer  is  neglected  for  a system  with  n buffers, 
then  Sp*Pp(  given  that  n buffers). 

The  following  relations  are  derived: 

Sn+i-l/[l+(n+l)<i/(XSp)]  forn<ci  4.13 

and  Sp+i«l/[l+c|i/(XSp)]  for  n2c  4.14 

With  Sq-1  as  the  initial  condition. 

The  above  analysis  assumes  that  once  the  packet  is  sent  out,  it  relieves  the 
buffer.  It  is  true  if  no  acknowledgement  is  required  or  the  response  of  the 
acknowledgement  is  almost  immediately.  In  the  following,  the  finite  response  time  of 
acknowledgement  is  considered.  Let  I be  the  number  of  output  links  of  this  integrated 
switch.  Those  packets  which  have  been  processed  by  this  integrated  switch  are  sent 
out  through  the  corresponding  links  to  the  next  switch.  Let  qg  and  Sg  be  the  queue 
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and  service  time  of  this  integrated  switch.  qj,..untit  qj  are  the  queues  corresponding  to 
the  links  through  which  packets  are  sent  out  and  waiting  for  acknowledgements.  The 
(1+1  )st  queue  is  the  pooling  queue  for  the  unoccupied  buffers.  The  service  rate  of 
(1+1  }st  queue  is  the  input  rate  of  this  integrated  switch.  The  probability  that  zero 
customer  in  (l+l)st  queue  is  the  probability  that  a packet  coming  into  the  integrated 
switch  is  neglected. 

For  exponential  response  time  of  acknowledgement,  the  (aordon  and  Newell’s 
result  of  product  form  is  used: 


1 1+1  n; 

, P^^o»^l»*"»'^|+ll“g(^jr^CWi)  /Aj(nj)]  4.15 

where  (Xo,X],..,X|^|)  is  a real  positive  solution  to  the  eigenvector-like  equations, 
1+1 

PjXj-.E^jXjPjj  Oijsd+l)  4.16 

Pjj  are  the  branching  probability  from  queue  i to  queue  j and  G(N)  is  a 
normalizing  constant  defined  so  that  all  P(ng,..nj^j)  sum  to  one.  That  is, 

1+1  n' 

A.17 

l+I 

where  S(N,M)-{(nQ,nj,..,n|+2)|.2r  nj-N  and  nj^O  Vi},  and  Aj(n)  is  defined 
recursively  as  follows: 

Ah(0)-1, 


A^(n)-n‘A|^(n-l) 
A^^(n)-C|^•  A^(n-l) 


f nzcK 


where  c^  is  the  number  of  server  in  service  center  k. 

Algorithm  derived  by  Buzen[Buz  73]  is  used.  Assuming  Xo,J(|.^l  are  solved.  Let 

Note  that  Ca(N)  is  equal  to  g(NJ+l),  and,  in  fact,  g(nJ+l)-(Kn)  for  n-0,l,2p.N. 


1 


From  the  above  equations  that 
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g(n,0)-Xo"/Ao(n)  for  n-0,l,2_,N. 

end  g{0,m>-l  for 

With  the  recursive  relation  of  g(n,m): 

^ k 

g(n,m)-^(X^r/A„(k)-g(n-k,n'-l) 

g(n,ffl)  can  be  calculated.  The  marginal  probability  pj^.^  is  calculated. 


p 


Figure  3 shows  the  result  of  the  above  algorithm.  Compare  to  the  result  of 
system  without  acknowledgement,  it  is  not  surprised  that  the  finite  response  time  of 
acknowledgement  gives  a higher  demand  of  the  buffer  space. 


There  are  N buffers  and  M queues  in  the  system.  There  are  well-known 
algorithm  to  solve  the  exponential  service  time.  Gordon  and  Newell  [Gord  67]  gave  the 
product  form  solution.  Buzen  [Buz  73]  gave  an  efficient  algorithm  to  solve  this  kind  of 
system.  For  non-exponential  service  time  , Chandy  et  al  [Chan  75]  gave  an 
approximation  algorithm  to  solve  this  kind  of  system.  The  approximation  is  exact  for 
exponential  service  time.  The  approximation  algorithm  uses  the  concept  of  Norton’s 
Theorem  for  electrical  circuit  theory.  The  algorithm  is  fully  elaborated  in  the  papers 
■of  [Chan  75].  A brief  introduction  is  given  here. 

Tfie  idea  of  complement  of  queue  is  used.  The  complement  of  a queue  i in  a 

e 

network  R is  a queue  which  completely  captures  the  interface  between  queue  I and  the 


rest  of  the  network.  For  queueing  network  of  exponential  service  time,  local  balance 
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is  satisfied  and  the  complement  of  queue  can  be  determined.  But  for  general  network 
(which  do  not  satisfy  local  balance),  the  complement  can  not  be  determined  exactly  and ' 
must  be  approximated.  The  complement  of  a queue  is  approximated  by  a queue  with 
independent  exponential  service  time;  the  service  rate  for  this  queue,  however,  is 
appropriately  adjusted  to  account  for  the  assumptions  of  independence  and 
exponentiaiity.  This  is  referred  by  Chandy  as  local  balance  interface,  since  it  is 
computed  by  using  assumptions  of  local  balance. 

The  complementary  algorithm  is  carried  out  in  iterative  steps  , sequentially 
adjusting  local  balance  interface  to  better  approximate  the  complementary  queues.  On 
the  kth  iteration  of  the  algorithm  k-0,1,2,...  local  balance  interfaces  are  calculated 
using  an  auxiliary  network  The  auxiliary  network  is  identical  to  the  actual 

network  except  that 

(i) the  service  rates  in  are  different  and 

(ii) S^*'^  is  assumed  satisfy  local  balance. 

It  is  therefore  possible  to  use  Norton’s  Theorem  to  compute  the  complement  of 
every  queue  in  At  the  kth  iteration  the  complement  of  queue  I in  is 

approximated  to  the  complement  of  queue  i in  R.  In  other  words,  the  complement  of 
queue  i in  is  used  as  the  local  balance  interface  of  queue  i in  R. 

The  two  networks  in  the  figure  are  equivalent  for  networks  which  satisfy  local 
balance.  As  in  the  figures,  the  reduced  network  of  queue  i becomes  a closed  tandem 
queue  of  queue  i and  its  complement.  The  service  of.  the  complement  queue  is 
exponentially  distributed  with  state  dependent  rate.  So  to  queue  i,  it  is  equivalent  a 
state  dependent  Poisson  input  with  an  arbitrary  service  time.  It  is  a M/G/l/Xj  queue. 
For  service  distribution  with  a rational  Laplace  transform,  there  is  an  embedded 


Markov  process.  Then  this  embedded  Markov  process  is  solved  as  in  the  following 
figure. 


Two  parameters  is  checked  to  verify  proper  interface,  mean  queue  length  and 

throughputs  of  each  node.  Let  Qj  be  the  queue  length  of  queue  i and  E[qj]  be  its  mean 

value,  tj  be  the  throughputfnumber  of  customers  served  per  unit  time)  of  queue  L The 

two  conditions  checked  are: 

M 

.E  iE[q:]-N  , the  total  number  of  customers  in  system. 

'd 

C jtjPijat:  , j-l,2r~,M,  throughput  into  a node  is  equal  to  the  throughput  out 

* ’J  I 

of  a node. 

where  Pjj  is  the  probability  a customer  branches  to  queue  j after  service  in 
queue  i. 
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! 

If  the  input  process  is  Poissonian,  then  the  service  time  of  the  pooled  buffer 

I 

j 

I queue  is  exponential  distributed. 

I 

I 
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3.1.  Reliability  Considerations 


3.1.1.  Introduction 

As  the  basic  components  of  digital  systems  become  more  and  more  reliable  over 
the  years,  systems  sizes  tend  to  grow  larger  and  larger.  Increasing  also  is  the 
packaging  density  in  an  integrated  circuit  chip.  This  leads  to  increase  in  complexity  of 
systems.  In  general,  the  more  complex  a system  the  less  its  reliability.  Thus  there 
seems  to  be  an  effect  of  diminising  returns.  Accurate  reliability  modeling  is  vital  to 
correct  prediction  of  reliability. 

It  is  a common  practico  in  reliability  modeling  to  divide  a system  under 
investigation  into  a number  of  subsystems  or  modules.  A judicious  partitioning  leads  to 
a set  of  modules  that  are  statistically  mutually  indenpedent.  The  reliability  of  the 
system  is  then  merely  the  product  of  reliabilities  of  various  modules. 

The  problem  that  still  remainr  is  that  of  finding  the  reliability  of  individual 
modules;  In  the  days  of  discrete  components  it  was  a fairly  well  accepted  practice  to 
assume  the  statistical  independence  at  the  gate  level,  and  raise  the  gate  reliability  to 
the  number  of  gates  in  the  module.  Given  the  gate  reliability,  this  yielded  a good 
estimate  for  the  module  reliability.  The  present  day  technology  of  large  scale 
Integration  renders  the  technique  obsolete.  Although  still  a function  of  the  number  of 
gates,  the  complexity  may  no  longer  be  treated  as  a linear  function  of  the  number  of 
gates.  The  following  section  outlines  the  currently  acceptable  approach. 
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3.1.2.  Parts  Count  Model 

Before  getting  to  the  parts  count  model,  let  us  review  the  basic  assumptions.  It 
will  be  assumed  the  system  is  constructed  of  printed  circuit  boards.  The  PC  boards 
hold  IC  chips.  It  is  assumed  that  the  system  may  be  partitioned  into  chips  and  that  the 
IC  chips  are  statistically  independent.  It  is  further  assumed  that  the  reliability  of  a 
single  module  is  exponentially  distributed  or  that  the  failures  of  a single  chip  follow 
Poisson  distribution.  In  other  words. 

Probability  of  k failures  in  time  interval  (0,t)  - e-^*  (Xt)*  / (k!) 

Reliability  > probability  of  no  failures  in  <0,t) 

-e-*‘ 

With  these  assumptions,  if  a system  does  not  contain  any  redundancy  (i.e.  every 
subsystem  must  function  properly  for  the  system  to  work),  the  system  reliability  is 
also  exponential  in  nature.  Furthermore,  the  failure  rate  of  the  system  is  the  sum  of 
failure  rates  of  individual  modules. 

To  estimate  the  failures  rates  of  chips  we  will  use  the  data  published  in  Military 
Standardization  Handbook  217-B  [Mil74].  The  handbook  suggests  the  following  model 
for  the  failure  rate  of  a single  chip. 

X - Hi  n?  { n»d,  G«  + n^d?  G*  ) 

where  d|,  d„  oC,  fi  and  n’s  are  various  constants, 

G is  the  number  of  gates  in  the  chip. 

The  n’s  are  constants  that  depend  on  various  factors,  such  as  environment 
(whether  the  system  is  ground  based,  fixed  or  spaceborne,  etc.),  expected  junction 
temperature,  quality  control  (how  rigorously  has  the  chip  been  subjected  to  tests  and 
burn-in).  Table  3.1  shows  the  failure  rates  for  chips  with  various  number  of  gates.  The 
system  is  assumed  to  be  ground  based,  fixed,  the  quality  control  factor  is  assumed  to 
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be  10  {which  is  in  between  the  MIL-STD  factor  of  1.0  and  the  factor  to  be  used  for 
minimal  quality  control,  150),  and  the  junction  temperature  is  assumed  to  be  50.  The 
other  constants,  d„  d;,  oC  and  fi,  are  fixed  and  equal  to  0.00129,  0.00389,  0.67  and 
0.35  respectively. 


Table  3.1 


! 

I 


f 

i 


Gates 

Failures 
in  10®  hours 

Gi>tes 

Failures 
in  10®  hours 

1 

0.04342 

9 

0.10559 

2 

0.05711 

10 

0.11037 

3 

0.06721 

11 

0.11490 

4 

0.07553 

12 

0.11920 

5 

0.08275 

13 

0.12332 

6 

0.08920 

14 

0.12728 

7 

0.09508 

15 

0.13108 

8 

0.10052 

16 

0.13475 

The  estimation  of  failure  rate  for  a module  is  best  described  by  an  example. 


3.1.3.  Example 


As  an  example  let  us  consider  the  Processor  Interface  module  in  C.mmp.  Figure 
3.1  shows  the  chip  lay-out  and  list  of  parts  for  the  Processor  Interface.  From  this  data 
we  form  the  Table  3.2. 

Table  3.2  Calculation  of  \ for  Processor  Interface 


No. 

Id 

Gates 

X 

1 

74S140 

4 

0.07553 

1 

7440 

4 

0.07553 

1 

7404 

6 

0.08920 

2 

74S138 

16 

0.13475 

6 

7438 

4 

0.07553 

13 

74S74 

10 

0.11037 

From  individual  Xs  in  the  table,  we  estimate  X for  the  Processor  Interface  to  be 
X - 0.07553  + 0.07553  ♦ 0.08920  ♦ 2(0.13475)  ♦ 6(0.07553)  ♦ 13(0.11037) 
- 2.39775  failures/10^  hours. 
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I Thus  we  arrive  at  the  failure  rate  for  Processor  Interface.  Similar  calculations 

I 

! are  carried  out  for  all  subsystems  of  C.mmp  to  yield  the  overall  failure  rate.  The 

I 

j following  table  shows  failure  rates  for  the  various  subsystems  of  C.mmp.  The 

I processor-memory  switch  is  seen  to  have  the  largest  failure  rate. 

I 

I 

Table  3.3  Failure  rates,  C.mmp 

Subsystem  X in  failures 

j per  million  hours 

1 1/40  Processor  57.5 

Processor  associated  ckts.  11.41 

Memory  associated  ckts.  7.14 

Processor-memory  switch  202.403 

16K-words  memory  (core)  34.0 

' The  following  section  will  compare  reliabilities  of  both  C.mmp  and  CM*  (Computer 

Modules)  in  both  non-redundant  and  redundant  configurations. 


i 

I 
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3.2.  Reliability  Comparison  of  C.mmp  and  Cms 


I Parts  count  reliability  models  were  developed  for  both  of  the  in-house  systems 

, at  CMU,  the  multiminiprocessor  C.mmp  and  the  computer  modules  system  Cm*.  As  the 

I ( 

I conceptual  diagrams  of  Figure  3.2  depict,  both  are  general  purpose  multiprocessor 

I 

1 

I systems.  C.mmp  is  a general  purpose  system  with  a fixed  architecture.  Cm*,  on  the 

I 

other  hand,  has  a flexible  architecture  that  may  be  so  modified  as  to  afford  optimal 
I performance  for  a given  application. 

I As  multiprocessor  systems,  both  C.mmp  and  Cm*  offer  potential  processing 

i 

power  well  beyond  that  of  a single  processor.  C.mmp  has  an  upper  limit  of  16 
processors  (since  the  existing  switch  has  only  16  ports  for  processors).  In  concept,  the 

I 

j Cm*  architecture  is  arbitrarily  extendible;  the  only  limiting  factors  are  the  cost  and 

I 

I fundamental  limits  of  algorithms  programmed.  The  immediate  goal  of  the  Cm*  project  is 

I 

I to  have  a single-cluster  8-processor  system  functional  by  January  1977.  When  ail  of 

I 

t 

t 

I the  processing  power  is  not  required,  it  is  possible  to  conceive  of  either  C.mmp  or  Cm* 

I as  a potentially  redundant  architecture.  If  a task  requires  the  processing  capability  of, 

i 

I for  example,  only  4 processors,  then  we  may  view  the  other  processors  as  stand-by 

spares.  Assuming  that  we  can  detect  and  locate  a faulty  component  (processor, 
j memory,  switch),  and  the  malfunction  was  not  an  irrecoverable  one,  we  can  then 

' logically  replace  a faulty  component  with  a stand-by  spare.  The  structures  are  thus 

fault -tolerant  and  have  greater  reliability. 

To  arrive  at  the  reliabilities  of  multiprocessor  fault -tolerant  systems,  we  need  to 
use  two  levels  of  modeling.  We  apply  the  parts  count  reliability  model  to  estimate  the 
' failure  rates  of  individual  components.  Then  using  the  reliabilities  of  non-redundant 

comporwnts,  we  model  the  fault-tolerant  system  to  arrive  at  a system  reliability. 
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1.  Parts  Count  Reliability  Model  : The  failure  rates  for  standard  IC  chips  are 
found  in  Military  Standardization  Handbook  (MIL-STD-HDBK-2 1 7B).  Assuming 
exponential  distribution  for  reliability  and  mutual  statistical  independence,  the  failure 
rate  of  each  component  is  estimated  as  the  sum  of  the  failure  rates  of  its  various 
subcomponents.  Since  the  handbook  also  predicts  the  failure  rates  for  such 
subcomponents  as  resistors  or  printed  boards,  completeness  of  the  model  is  assured. 
The  following  are  the  failure  rates  for  various  components  of  C.mmp  and  Cmt  systems 
using  the  parts  count  reliability  model. 

Component  Failure  rate 

(failures  per  10®  hrs.) 


C.mmp 

POP- 11/40 

57.496 

Processor  associated  circuitry 
(RELOC,  processor  interface) 

11.414 

Memory  box  (16K  words;  core) 

54.225 

Memory  associated  circuitry 
(Priority  decode,  etc.)  per  port 

7.14 

Switch 

202.403 

Cm* 

LSI- 11  processor 

109.0 

Memory  (12K  words;  semiconductor) 

203.343 

K.map 

178.414 

2.  Redundancy  Model  : In  the  absence  of  realistic  data  on  detection  and 
replacement  capabilities  in  multiprocessor  systems,  we  use  the  following  simplistic 
model  (giving  us  the  upper  bound  on  potential  reliability).  If  there  are  N identical 
components  with  the  reliability  of  each  component  R,,  (Rg  - e~^t  where  X > failure 
rate),  and  if  a task  requires  k components,  the  subsystem  can  tolerate  upto  N-k 
failures,  and  the  reliability  of  such  a subsystem  is 

Z Cfl  R?-'  (1  - Ro)' 

Thus  the  reliability  of  C.mmp  with  16  processors  and  16  64K-memories,  with  at 
least  4 processors  and  4 memory  ports  required  for  the  task,  is 
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where  R,  > switch  reliability 

Rf  - (processor  + associated  circuitry)  reliability 
R,  ■ (memory  + associated  circuitry)  reliability. 

A similar  approach  is  applicable  to  Cm*  system.  The  reliability  of  a single  cluster 

8-processor  Cm*,  for  a tasK  requiring  4 processors  and  4 memories,  is 

R*.(  RcV  (1  -Rc»)'  ) 

where  R^a  - reliability  of  (K.map  + map  bus) 

Rck  ~ reliability  of  a computer  module 

- product  of  reliabilities  of  S.local,  memory 
and  processor. 

The  reliabilities  of  the  two  redundant  systems  described  by  the  two  equations 
above  are  calculated  and  are  presented  in  Figures  3.3  and  3.4.  Figure  3.3  shows  the 
reliabilities  of  a 16-processor  C.mmp  as  a function  of  time  for  tasks  requiring  various 
number  of  processors.  The  plot  for  task  processor  - 16  is  the  reliability  of  a totally 
non-redundant  Crnmp.  The  dramatic  increase  in  reliability  as  the  number  of  task 
processors  decreases  is  evident.  Figure  3.4  depicts  similar  plots  for  a single  cluster 
Cm*  system.  Again  there  is  noticeable  improvement  in  reliability  as  we  move  from  a 
non-redundant  to  redundant  system. 

To  compare  the  two  systems,  we  use  the  notion  of  mission  lime  improvement 
(MTI)  [Kno64].  If  the  minimum  reliability  desired  is  Rsvtntm  mission  lime  tm  of  a 
system  is  defined  by  Rsvs(tm)  * RiwtMin-  The  MTI  of  a system  1 over  a system  2 is  then 
defined  to  be  tmi/tm;.  It  is  the  measure  of  how  much  longer  the  system  1 is  able  to 
keep  its  reliability  above  the  minimum  required  than  the  system  2.  To  compare  the 
two  systems  C.mmp  and  Cm*,  we  will  use  LSI- 11  as  a base.  We  find  the  MTI  of  C.mmp 
over  LSI-11  by  t,/tj  where  Rc.Mtfdt)  ■ Rut-nd?).  Since  the  MTI  is  not  constant  over  all 
values  of  RivMim  plot  these  as  a function  of  R|,«vti  'o  Figure  3.5.  A similar  graph 
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is  plotted  for  MTI  of  Cm*  over  LSI- 11  in  Figure  3.6.  In  order  to  Keep  the  comparison 
fair,  the  memory  boxes  for  C.mmp  were  normalized  to  16K  words  in  Figure  3.5. 

The  plots  of  MTI  show  the  16-processor  C.mmp  to  be  slightly  less  reliable  than 
a single  cluster  8-processor  Cm*.  The  chief  reason  for  this  is  that  although  C.mmp  has 
more  processors  and  allows  greater  redundancy,  the  non-redundant  part,  the  switch,  is 
considerably  less  reliable  than  the  corresponding  Cm*  component,  K.map.  When  Cm* 
achieves  its  more  distant  goals  of  having  a multi-cluster  system,  normalization  of 
processing  power,  memory  capacity,  and  redundancy  will  be  required. 

Up  to  this  point,  we  have  been  discussing  the  reliabilities  of  both  non-redundant 
and  redundant  systems  in  a long  mission  mode.  No  repairs  were  presumed.  However,  in 
systems  such  as  the  integrated  communications  switch,  periodic  maintenance  is  not  only 
possible  but  is  also  highly  advisable.  The  following  treatment  discusses  the  effect  of 
maintenance  on  the  system  reliability. 


-! 
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3.3.  Effect  of  Periodic  Maintenance  on  Reliability 


3.3.1.  Introduction 
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In  an  attempt  to  increase  the  life  of  a non-perfect  system,  various  redundancy 
techniques  have  been  applied.  It  has  been  shown  that  TMR  (triple  modular  redundancy) 
with  stand-by  sparing  can  be  used  to  achieve  improvement  in  reliability  and  expected 
life.  While  the  general  analysis  holds  perfectly  well  for  systems  performing  vital 
functions  in  a space  mission,  it  is  extremely  pessimistic  in  a more  commonly  used 
system  where  a technician  may  be  able  to  perform  repairs.  Even  more  common  is  a 
situation  in  which  certain  tests  are  applied  to  the  system  at  regular  intervals  to  insure 
its  integrity,  followed  by  any  equired  ’■epair.  It  is  to  be  expected  in  most  cases  that 
the  life  span  of  a system  subjected  to  such  maintenance  would  be  greater  than  that  of 
an  identical  system  left  unattended.  In  the  following  analysis  we  estimate  the  expected 
life  of  a non-perfect  system  under  periodic  maintenance. 

3.3.2.  Life  of  An  Unmaintained  System 

As  has  been  the  common  practice,  we  will  assume  the  failures  in  a non- 
redundant  system  to  have  an  exponential  distribution.  We  will  denote  the  failure  rate 
by  X.  The  reliability  of  the  system  (i.e.  the  probability  that  there  is  no  failure  during 
the  time  interval  (0,t))  is  then  R(t)  ■ e-^L  The  life  of  the  system  is  estimated  as 
follows. 

Let  T be  the  time  at  which  a failure  occurs.  Then  the  distribution  function 
F (t)  is  s 

F (t)  ■ Prob  (the  failure  occuring  in  (0,1)) 


I 
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- Prob(0<r<t) 

- 1 - Prob(r>t) 

- 1 - R(t) 

The  life  of  the  system  is  the  expected  value  of  T,  EfD. 

«o 

And,  E(r)  » / (1  - F(t))  dt 

r • 

- ^ R(t)  dt 

Note  that  the  above  equation  is  a general  one,  and  may  be  applied  to  any 
function  R(t).  Also  note  that  according  to  the  equation  above,  E(D,  or  the  life  of  the 
system,  is  the  same  as  the  area  under  the  curve  for  any  function  R(t).  For  the 
exponential  distribution, 

1 

E(D  - l/X. 

3.3.3.  Life  of  A Maintained  Systerri 
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Let  us  now  consider  a system  being  operated  under  periodic  maintenance.  We 
assume  that  certain  tests  are  performed  at  a regular  time  interval,  fi.  Every  occasion 
on  which  these  tests  are  performed  with  success  (or  the  failures  of  the  tests  are 
followed  by  subsequent  necessary  repairs),  there  is  a lesser  probability  of  failure  in 
immediate  future.  This  leads  to  an  increase  in  reliability  after  every  maintenance 
period.  We  shall  consider  two  models  for  the  improvement  obtained  by  periodic 
maintenance. 

Model  I : Improvement  through  maintenance  ■ d (1  - e-^*),  where  d is  a constant, 
O^d^l.  This  model  assumes  that  a constant  fraction  of  the  lost  reliability  is  recovered. 
Figure  3.7  shows  a hypothetical  reliability  function  using  this  assumption. 

Let  R|(t)  > reliability  function  during  the  ith  interval. 

Then, 

3'10 


I 

I 


f 


k 


Carnegie-N^llon  University  Report  June  26,  1976 

R,(t)  - e-^t 

R,(t)  - { e-^»  + d (1  - e-^») } e-^‘ 

Rftt)  - { e-2^*  + d (1  - e-^»)  e-«  ♦ d (1  - e-^*) } e-^» 

- { e-2^»  ♦ d (1  - e-2>'») } e-^‘ 

in  general, 

Rui<t)  - { e-'^*  + d (1  - e-'J'*) } e-^» 

The  area  under  the  (i+l)st  segment, 

Au,  - / Ru.(t)  dt 

- { e-'^»  + d (1  - e-'^»)  } e-^‘  dt 

- (1/X)  (1  - e-^*)  { e-'^»  + d (1  - e-'^*)  } 

The  attempt  to  evaluate  life,  which  is  also  the  sum  of  all  A|*s,  yields  for  an 
answer.  This  is  due  to  the  fact  that  under  the  assumptions,  the  reliability  function 
reaches  a steady  state  as  shown  in  Figure  3.8.  The  area  under  this  reliability  function 
is  not  finite. 

Since  the  tests  will  not,  in  general,  test  all  possible  system  components,  there 
will  remain  components  that  are  never  replaced  or  repaired  during  the  periodic 
maintenance.  These  components  will  eventually  fail.  Thus  an  expected  infinite  life  span 
is  unrealistic. 

Model  II  : In  the  first  model,  we  assumed  the  improvement  due  to  maintenance  to  be  a 
constant.  In  a more  pessimistic  model,  we  may  assume  that  the  improvement  in 
reliability  due  to  maintenance  at  the  end  of  the  ith  period  is  a fraction  of  the  reliability 
lost  in  the  ith  period.  In  other  words, 

Rui(t)  - { Ri,  e-^*  ♦ d R,o  (1  - e-^») } e-^‘ 

where  R|o  R|  at  beginning  of  ith  period. 

Writing  R/s  explicitly. 
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R,(t)  - e-^t 

i R,(t)  - { e-^»  + d (1  - e-^») } e-^‘ 

i 

I Rs(0  - [ {e-^»+d(l-e->'»)}e->‘»  + d{e-^»+d(l-e-^*)}(l-e-^*)  ] e->‘‘ 

I 

- { e-^»  + d (1  - e-^*)  }2  e-^» 

I 

and 

I 

I Rui(t)  - { e-^»  + d (1  - e-’'*)  )'  e-^< 

The  area  under  the  ith  segment, 

Au,  - { e-^*  + d (1  - e-^»)  )•  ^e->'<  dt 
j -(1/X)(1  -e-^»){e>>»  + d(l -e-^»))' 

^ And  now  the  expected  life  is  given  by 

life  - E Ai+, 

- (1/X)  (1  - e-^*)  { I - e-^*  - d (1  - e-^»)  }-» 

i 

I .l/{X(l-d)} 

! The  life  span  is  thus  improved  by  a factor  of  (1  - d)-*.  If  d ■ 1,  the  maintenance 

I 

I is  perfect  and  the  life  span  tends  to  be  infinite.  If  d - 0,  we  revert  back  to  an 

I 

i unmaintained  system  with  life  span  (1/X). 

I 

t 

, 3.3.4.  Redundant  Systems  With  Maintenance 

; Until  now  we  have  considered  a non-redundant  system  with  R(t)  ■ e-^*.  Where 

reliabilities  of  high  order  (better  than  1 failure  in  a million  hours)  are  required, 
improvements  through  technological  advances  alone  fall  short  of  the  objective.  System 
designers  have  resorted  to  redundancy  to  attain  higher  reliabilities.  Triple  modular 
I redundancy  (TMR)  with  majority  voting  is  one  of  the  redundancy  techniques  used. 
Since  such  a technique  allows  the  system  to  tolerate  a single  failure  in  any  of  the 
three  modules,  the  reliability  of  a TMR  system  is 
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R,M(t)  - R*(t)  + 3 R2(t)  (1  - R(t)) 

- 3 R2(t)  - 2 R»(t) 

where  R(t)  > reliability  of  non-redundant  system. 

Since  we  intend  to  keep  the  development  applicable  in  general,  we  will  use 
Rsus(t)  during  the  discussion  and  substitute  the  expression  for  a TMR  system  only  to 
exemplify  the  results. 

3.3.5.  Life  of  An  Unmaintained.  Redundant  System 

Recalling  that  the  life  span  of  a system,  E(D,  is  the  same  as  that  of  the  area 
under  the  reliability  function  we  can  write 
Life  - ^ R„,(t)  dt 
For  a TMR  system, 

Life  (TMR)  = / { 3 R2(t)  - 2 R^t) ) dt 
-(lA)(3/2-2/3) 

- 5 / (6  X) 

Again,  we  now  focus  our  attention  to  a system  under  periodic  maintenarKe. 

, 3.3.6.  Life  of  A Maintained.  Redundant  System 

We  will  again  consider  the  two  models.  Under  the  first  simple  model,  we  assume 
that  the  improvement  through  maintenance  is  a constant  fraction  of  the  probability  of 
failure,  i.e.  d (1  - Rtvs(^))  where  (I  is  the  period  of  the  maintenance  cycle. 

Let  R|(t)  - system  reliability  function  during  the  ith  interval. 

Then, 

R,(t)  - R,v,(t) 
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Rr(t)  - { Rtv.(/8)  + d (1  - R,„(/l) ) R,„(t) 

R*(t)  - { Rf‘v.</3)  ♦ d (1  - R,„(/}»  R,v.(/3)  ♦ d (1  - R,v.(/J)) } R.v.(t) 
- { Ri;.(/8)  + d (1  - r'„(/3)  } R„.(t) 

And, 

Rt(t)  - { Ri«(/3)  + d (1  - Ri„(^)  } R.„(t) 

The  area  under  the  (i-t-l)st  segment  is 


- Ri«(/3)  + d (1  - R.‘„(/?) } R,„(t)  Jt 

- { Rmi</8)  + d (1  - r;„(^)  1 dt 


For  a TMR  system, 

R,v,(t)  - 3 e-2^»  - 2 e-»^* 


/ Rt«(»)  dt 

^ A 

- I { 3 e-2^t  - 2 e-»>'»  } dt 

- (3  / 2X)  (1  - e-2^*)  - (2  / 3X)  ( 1 - e->>'») 

We  again  note  that  when  we  try  to  sum  all  A(*s,  the  results  approaches 
Cte'dering  the  second  model  that  we  suggested  earlier,  where  the  improvement 
in  reliability  is  a fraction  of  the  reliability  lost  in  the  ith  period,  we  may  write 
R|4i(t)  - { R|(/J)  ♦ d R,(0)  (1  - R,«,(/J))  } R,v*(t) 

Writing  R|’s  explicitly,  we  have 
R|<t)  - R.vt(t) 

Ri«)  - I R.vs(/l)  ♦ d (1  - R.„(/J)) ) R.v.(l) 

R»<»)  - { Rw(/S)  ♦ d R,v.(/3)  (l-R.„(/?))  ♦ d R,v,(/3)  ♦ d2  (l-R„,(/J) 

- d R,»,</J)  - d2  R„,(/3)  (1  - R„,(/3)) } R,*.(t) 

- { Rsvt(/J)  ♦ d (I  - R,*,(/3))  }2  R,v,(t) 

And,  in  general. 
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Rui(t)  - { R.«.(/3)  + d (1  - R,„(/S)) }'  R„,{t) 

The  area  under  the  (i-*-l)st  segment, 

A,*,  - / R,*,(t)  d\ 

- { R.w.</8)  + d (1  - R„,(/3))  }'  ] R„,(t)  dt 

* 

Then  the  life  span  for  the  system  is 
Life  - E Au, 

- ( I R,v.(t)  dt ) E { R,«(/3)  + d (1  - R,w.(/J)) }' 

- < |^R.v.(t)  dt  ) { 1 - R„,(/!J)  - d (1  - R„,(/3)  }-l 

- ( /'*R.v.(t)  dt)  (1  - R,„(^))-‘  (1  - d)-» 

For  a TMR  system, 

R,v.(t)  dt  - (3  / 2X)  (1  - e-2^>)  - (2  / 3X)  (1  - e-»^>> 
and  R,w,(/a)  - 3 e-2^»  - 2 e-^*. 

Substituting  these  two  expressions  in  the  equation  for  Life  we  can  estimate  the 
life  span  of  a TMR  system.  As  the  expression  is  very  complex,  the  improvement  is  not 
obvious  in  this  form.  Let  us,  therefore,  consider  numerical  examples.  The  following 
table  (Table  3.4)  allows  the  comparison  in  the  life  spans  of  unmaintained  and 
maintained  TMR  systems. 

The  data  conform  to  the  expectations.  As  the  period  maintenance  becomes 
larger,  the  expected  life  becomes  shorter.  We  note  that  after  fi  ■ 10000,  the  life  is 
shorter  than  the  period  of  maintenance.  Thus  the  maintenance  has  no  effect  on  the 
performance  for  any  period  larger  than  10000.  The  life  depends  more  strongly  on  d, 
as  seen  by  the  sharp  increase  of  life  as  d approaches  unity.  The  constant  d Is  a 
function  of  how  efficient  maintenance  routines  are.  Since  the  expected  life  is  extremely 
sensitive  to  the  effectiveness  of  maintenance  around  d - 0.9,  slight  improvement  in 
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Table  3.4  Expaelad  Ufa  ulth  parledle  aainlanancai  Laada  » I.NIIM 
Lila  el  an  unaaintalnad  THR  systaa  i 8333.33 


Lila  el  a Maintained  eystaa  ■ 
d beta 


18 

180 

1888 

18888 

188888 

1888888 

0.18 

3789894.97 

376558.34 

43282.55 

10629.76 

9259.26 

9259.26 

8.28 

4173631.84 

423619.13 

48692.87 

11958.48 

10416.67 

10416.67 

8.38 

4769864.97 

484136.15 

55648.99 

13666.83 

11904.76 

11904.76 

8.48 

5564842.45 

564825.50 

64923.83 

15944.64 

13888.89 

13888.89 

8.se 

6677818.96 

677798.60 

77908.59 

19133.57 

16666.67 

16666.67 

8.68 

8347263.73 

847238.26 

97385.74 

23916.96 

20833.33 

20833.33 

8.78 

11129684.76 

1129650.99 

129847.65 

31889.28 

27777.78 

27777.78 

8.88 

16694527.14 

1694476.49 

194771.48 

47833.92 

41666.67 

41666.67 

8.98 

33389854.29 

3388952.97 

389542.96 

95667.84 

83333.33 

83333.33 

8.98 

33389054.29 

3388952.97 

389542.96 

95667.84 

83333.33 

83333.33 

8.9X 

37098948.28 

3765503.21 

432825.50 

186297.59 

92592.59 

92592.59 

8.92 

41736319.41 

4236191.38 

486928.72 

119584.80 

184166.67 

184166.67 

8.93 

47698649.49 

4841361.44 

556489.95 

136668.34 

119847.62 

119047.62 

8.94 

55648422.43 

5648254.82 

649238.25 

159446.39 

138888.88 

136888.88 

8.95 

66778183.59 

6777905.44 

779085.86 

191335.66 

166666.65 

166666.65 

8.96 

83472638.82 

8472382.75 

973857.43 

239169.60 

208333.34 

288333.34 

8.97 

111296844.85 

11296509.63 

1298476.49 

318892.78 

277777.77 

277777.77 

8.98 

166945246.55 

16944762.34 

1947714.50 

478339. 11 

416666.68 

416666.60 

8.99 

333890617.48 

33889537.31 

3895438.45 

956678.58 

633333.51 

833333.51 

maintenance  routines  may  be  rewarded  with  substantially  greater  life  for  the  system. 
This  fact  is  emphasized  when  we  plot  the  expected  life  of  a TMR  system  as  a function 
in  Figure  3.9. 
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FIGURE  3.1  Processor  Interface 


F1s^.'3.3:  Reliability  of  C.nmp  for  a task  requiring  K processors. 


0.5  Reliability  of  LSI- 11 
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Figure  3.7  : Effect  of  periodic  maintenance  on 
system  reliability. 
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Figure  3.8  : Steady  state  reached  by  the  reliability 
function  in  the  first  model  for  periodic 
maintenance. 
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